Characteristic offset-based occupying object detection method and device

Through the detection method based on feature offset, the accuracy and accuracy of the occupancy object recognition are improved, the problem of insufficient accuracy of the identification method in the prior art is solved, and pipeline safety management is enhanced.

CN119007119APending Publication Date: 2024-11-22山东智航智能装备有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411239643.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The accuracy and accuracy of the occupancy identification method in the prior art has low accuracy and accuracy, which has affected pipeline safety and operation.

Method used

Using a detection method based on feature offset, the target object is extracted from the image sequence, detected feature points, matched and offset compensation of feature points, clustered static target objects, judged whether it is in a threat area, and iteratedly performed the classification steps to Improve accuracy.

Benefits of technology

It improves the accuracy of target object classification and the accuracy of judging of pressure objects, reduces the harm caused by pipeline pressure, and facilitates pipeline management and maintenance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007119A_ABST
    Figure CN119007119A_ABST
Patent Text Reader

Abstract

The invention discloses a feature offset-based occupying object detection method and device. The method comprises the following steps of: detecting feature points in an image sequence set; iteratively executing the classification step until a preset iteration frequency is reached, and judging that the static target object is an occupying object; a classification step: performing feature point matching based on the feature point coordinates, determining the offset of the target image, and performing offset compensation to obtain a compensation offset; clustering the compensation offset, and dividing the target object into a static target object and a dynamic target object; judging whether the static target object is in the threat area or not, and if not, determining that the static target object is not an occupying object; and if yes, taking the image of the previous frame of the target image where the static target object is located as the target image to execute a classification step. The problem that an existing occupying object identification method is low in accuracy and precision is solved. The target classification precision and the occupying object identification accuracy can be improved, management and maintenance of managers can be facilitated, and the pipeline occupying damage is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of gas pipeline network risk management and control, and in particular to a method and device for detecting occupied objects based on characteristic offsets. Background Art

[0002] With the rapid economic development and the expansion of urban and rural planning, the problem of illegal occupation of various oil and gas pipelines has become increasingly prominent. Illegal occupation of pipelines refers to the construction and installation of buildings (structures) and other objects above the pipeline that endanger the safety of the pipeline and affect the operation and maintenance of the pipeline. Occupation will have different degrees of impact on the pipeline itself, pipeline corrosion protection, pipeline maintenance and repair, and building (structure) safety, so the cleaning and control of pipeline occupation is particularly important.

[0003] The main manifestations of pipeline occupation are: 1. Buildings or overweight vehicles occupying pipelines cause underground pipelines to sink and deform, resulting in pipeline rupture and oil and gas leakage; 2. Waste liquid generated by occupied buildings seeps into the ground, accelerating pipeline corrosion and perforation; 3. Due to pipeline occupation, leaked gas cannot be removed in time, and explosion is prone to occur when it accumulates to a certain concentration; 4. Occupied buildings and vehicles are used to carry out illegal and criminal activities such as stealing oil and gas; 5. Occupied pipelines cannot be maintained and inspected normally, and inspection personnel cannot directly observe the basic conditions of the pipelines, which affects the inspection of special equipment. When dangerous situations occur, it is difficult for maintenance equipment to enter the site, delaying the time for emergency repairs.

[0004] However, in the prior art, the identification of occupied objects is performed through steps such as target recognition and feature point matching on images. The identified occupied objects often include temporary targets (such as temporary parking), and the identification accuracy and precision are low. Summary of the invention

[0005] The embodiments of the present application solve the problem of low accuracy and precision of occupied object identification methods in the prior art by providing an occupied object detection method and device based on feature offset.

[0006] In a first aspect, an embodiment of the present application provides an occupier object detection method based on feature offset, comprising: extracting a target object on a target image from an image sequence set; detecting feature points in the image sequence set; iteratively performing a classification step until a preset number of iterations is reached, and then determining that a static target object is an occupier object; the classification step comprises: performing feature point matching on the target image and its previous frame in the image sequence set based on the feature points; determining the offset of the target image based on the feature point matching result, and performing offset compensation on the offset to obtain a compensated offset; clustering the compensated offset, and dividing the target object into a static target object and a dynamic target object according to the clustering result; judging whether the static target object is in a threat area, and if the static target object is not in the threat area, determining that the static target object is not an occupier; if the static target object is in the threat area, performing the classification step using the image of the previous frame of the target image where the static target object is located as the target image.

[0007] In combination with the first aspect, in a possible implementation method, the offset compensation for the offset to obtain the compensated offset includes: obtaining the navigation state and shooting roll angle of the drone when shooting the target image; selecting an offset compensation method according to the navigation state and the shooting roll angle to determine the compensated offset; wherein the offset compensation method includes perspective transformation offset compensation and / or moving mean offset compensation.

[0008] In combination with the first aspect, in a possible implementation method, the offset compensation method is selected according to the navigation state and the shooting roll angle, including: when the navigation state of the UAV is a straight navigation state and the shooting roll angle is within a first angle range, the offset compensation method is the perspective transformation offset compensation; when the navigation state of the UAV is a straight navigation state and the shooting roll angle exceeds the first angle range, the offset compensation method is to first perform the perspective transformation offset compensation and then perform the moving mean offset compensation; when the navigation state of the UAV is a turning state, the offset compensation method is the moving mean offset compensation.

[0009] In combination with the first aspect, in a possible implementation, the perspective transformation offset compensation includes: constructing a compensation table according to the imaging law of the target image; wherein the compensation table includes an X-direction compensation table and a Y-direction compensation table; setting the value of a row vector in each row according to the height of the X-direction compensation table; filling the value of a column vector in each column according to the data distribution of the offset in the Y-direction compensation table; determining the perspective transformation matrix of the target image based on the value of the row vector and the value of the column vector, performing perspective transformation on the compensation table through the perspective transformation matrix, and normalizing it to obtain a perspective transformation compensation table; determining the compensation coefficient of the corresponding feature point in the perspective transformation compensation table according to the feature point coordinates of the feature point, and determining the compensation offset of the feature point through the compensation coefficient.

[0010] In combination with the first aspect, in a possible implementation, setting the value of the row vector in each row according to the height of the X-direction compensation table includes: setting the value of the middle position row vector of each row in the X-direction compensation table to half the height of the X-direction compensation table; taking the middle position row vector as a reference, setting the values ​​of the row vectors within the preset ranges at both ends thereof in turn in arithmetic progression with a first difference as a tolerance, and setting the values ​​of the row vectors outside the preset range to the values ​​of the row vectors within the preset range closest to them; wherein the preset range is determined according to the width of the X-direction compensation table, the row number of the middle position row vector, and the angle between the drone and the pod camera when shooting the target image.

[0011] In combination with the first aspect, in a possible implementation method, the value of the column vector in each column is filled according to the data distribution of the offset in the Y-direction compensation table, including: if the offset of the target image decreases from bottom to top, the value of the top column vector in the Y-direction compensation table is set to half the height of the Y-direction compensation table, and the values ​​of the remaining column vectors are set arithmetic differences from top to bottom with a second difference as a tolerance; if the offset of the target image increases from bottom to top, the value of the bottom column vector in the Y-direction compensation table is set to half the height of the Y-direction compensation table, and the values ​​of the remaining column vectors are set arithmetic differences from bottom to top with the second difference as a tolerance; wherein the second difference is determined according to the angle between the drone and the pod camera when shooting the target image; if a negative value appears when setting the value of the column vector arithmetic differences, it is set to 0.

[0012] In combination with the first aspect, in a possible implementation manner, the moving mean offset compensation includes: determining an X offset / Y offset of the feature point in an X direction / Y direction row by row / column based on the offset of the target image; determining a sum of X offsets / a sum of Y offsets according to the X offset / the Y offset, and filtering the sum to obtain a row offset / column offset of a corresponding row / column; and determining the compensation offset according to the row offset and the column offset.

[0013] In a second aspect, an embodiment of the present application provides an occupier detection device based on feature offset, comprising: an extraction module, used to extract a target object on a target image from an image sequence set; a detection module, used to detect feature points in the image sequence set; an iteration module, used to iteratively execute a classification step until a preset number of iterations is reached, and then determine that the static target object is an occupier; the classification step comprises: performing feature point matching on the target image and its previous frame in the image sequence set based on the feature points; determining the offset of the target image based on the feature point matching result, and performing offset compensation on the offset to obtain a compensated offset; clustering the compensated offset, and dividing the target object into a static target object and a dynamic target object according to the clustering result; judging whether the static target object is in a threat area, and if the static target object is not in the threat area, determining that the static target object is not an occupier; if the static target object is in the threat area, performing the classification step using the image of the previous frame of the target image where the static target object is located as the target image.

[0014] In a third aspect, an embodiment of the present application provides a device, comprising: a processor; a memory for storing processor-executable instructions; when the processor executes the executable instructions, it implements the method described in the first aspect or any possible implementation method of the first aspect.

[0015] In a fourth aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium includes a medium for storing a computer program or instructions, which, when executed, enables the method described in the first aspect or any possible implementation method of the first aspect to be implemented.

[0016] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: The embodiment of the present application can determine the compensation offset of the feature points in two adjacent frames of images by matching and compensating for the offset, thereby improving the accuracy of target classification; by clustering the compensation offset, the state of the target can be determined; by iteratively executing the classification step, the accuracy of the occupied object judgment can be improved, effectively solving the problem of low accuracy and precision of the occupied object identification method in the prior art. The image quality can be improved, and the accuracy of target classification can be improved. By performing multiple iterative judgments and finally determining the occupied object, the accuracy of occupied object identification can be improved. By identifying the occupied object, it is convenient for management personnel to perform daily management and maintenance of the pipeline, and reduce the harm caused by pipeline occupation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of an occupied object detection method based on characteristic offset provided in an embodiment of the present application; Figure 2 A flow chart of a method for selecting an offset compensation method according to a navigation state and a shooting roll angle provided in an embodiment of the present application; Figure 3 A flow chart of a method for perspective transformation offset compensation provided in an embodiment of the present application; Figure 4 A flow chart of a method for compensating a moving mean offset provided in an embodiment of the present application; Figure 5 A flow chart of dividing targets into static targets and dynamic targets according to clustering results provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of an occupied object detection device based on characteristic offset provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of a drone equipped with a camera for shooting images provided in an embodiment of the present application; Figure 8 An example diagram of the coordinates of corner points before and after perspective transformation of the compensation table provided in the embodiment of the present application; Fig. 9 An example diagram of a static target, an occupied object, and a threat area provided in an embodiment of the present application; Fig.10 An example diagram of the division results of static feature points and dynamic feature points provided in an embodiment of the present application; Fig.11 An example diagram of a non-permanent static target provided in an embodiment of the present application; Fig.12 This is an example diagram of a permanent static target object provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The following describes some of the techniques involved in the embodiments of the present application to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted in the following description.

[0021] Figure 1 1 is a flowchart of an occupied object detection method based on characteristic offset provided in an embodiment of the present application, including steps 101 to 110, as follows.

[0022] Step 101: extracting the target object on the target image from the image sequence set. In the embodiment of the present application, the images taken by the drone during the pipeline inspection are obtained, and these images are stored as an image sequence set in the order of shooting. The current image in the image sequence set is used as the target image, and the target object therein is extracted.

[0023] Exemplarily, the target object in the target image may be a building, a car, a truck, a bus, a construction vehicle, a tanker, a two-wheeled vehicle, a bicycle, a person, etc. The target object that needs to be determined as an occupied object is a building, a car, a truck, a construction vehicle, a tanker, a bus, etc. In this application, the YOLOv5 (full name: You Only Look Once version 5, a single-stage target detection algorithm, one of the YOLO series algorithms) algorithm may be used to extract the target object.

[0024] Step 102: Detect feature points in the image sequence set. In the embodiment of the present application, feature points refer to pixel points at key positions in the image sequence set that are prominent and easy to identify, and these pixel points exist continuously and stably in multiple images in the image sequence set, and can be reliably detected and matched under different viewing angles, lighting conditions or other changes, such as corner points, edge points, texture feature points, etc. of the image.

[0025] Exemplarily, the goodFeaturesToTrack() function (a function in the OpenCV library for detecting strong corners in an image) is used here to perform feature point (corner point) detection. Those skilled in the art can also extract feature points and determine feature point coordinates by methods such as the Harris algorithm (a corner point detection algorithm), the SIFT algorithm (English full name: Scale-Invariant Feature Transform, Chinese: Scale-Invariant Feature Transform), and the SURF algorithm (English full name: Speeded-Up Robust Features, a fast and robust feature point detection and description algorithm).

[0026] Step 103: Based on the feature points, feature point matching is performed on the target image and the image of the previous frame in the image sequence set. Exemplarily, the sparse optical flow method is used here for feature point matching. Those skilled in the art may also use feature point matching algorithms such as brute force matching algorithm, nearest neighbor search algorithm, etc.

[0027] Step 104: Determine the offset of the target image based on the feature point matching result, and perform offset compensation on the offset to obtain a compensated offset. In the embodiment of the present application, the feature point matching result is a feature point pair. The offset of the target image and the image of the previous frame is determined by calculating the coordinate difference between the feature point coordinates of the feature point pair.

[0028] Specifically, the feature point pairs in the feature point matching result are exemplarily recorded as and ,in, represents the feature points in the target image, Indicates the image of the previous frame of the target image. Corresponding feature points. The offset of the corresponding target image is obtained by calculating the coordinate difference between the feature point pairs, as follows: In the formula, express The offset of the target image where the point is located, express The coordinates of express The coordinates of .

[0029] In the embodiment of the present application, during the drone pipeline inspection, due to various interference factors, the offset of the static feature points obtained from two consecutive frames of images has a large deviation. In the actual occupier judgment, the offset of the static feature points should be basically consistent. In order to ensure the validity of the subsequent calculation results and reduce the judgment error, the present application performs offset compensation on the offset of the feature points to make it conform to the original distribution law.

[0030] The steps of performing offset compensation on the offset to obtain the compensated offset are as follows: Figure 2 As shown, it includes steps 201 to 207, which are specifically as follows.

[0031] Step 201: Obtain the navigation state and shooting roll angle of the drone when shooting the target image. Specifically, obtain the navigation state and shooting roll angle of the drone when shooting the target image from the flight control system of the drone or the ground station software. The navigation state here refers to whether the drone is in a straight-line navigation state or a turning state. Figure 7 As shown, the X-axis direction in the figure represents the direction of the drone's nose. Indicates the angle between the drone and the pod camera.

[0032] Step 202: Determine whether the navigation state is a straight navigation state. Specifically, if the navigation state of the drone is in a turning state when shooting the target image, execute step 203. If the navigation state of the drone is in a straight navigation state when shooting the target image, execute step 204.

[0033] Step 203: The offset compensation method is moving mean offset compensation. Figure 4 FIG. 1 is a flow chart of the moving mean offset compensation method of the present application. In the present application, the X offset / Y offset of the feature point in the X direction / Y direction is determined row by row / column based on the offset of the target image. Specifically, according to the feature point of the target image, The feature points corresponding to the previous frame of the target image The offset Determine the X offset of the feature point in the X direction row by row The details are as follows: In the formula, Indicates the X offset of the feature point with row number i and column number j on the target image. Indicates the feature point with row number i and column number j on the target image The horizontal axis of Indicates the feature point with row number i and column number j in the image of the previous frame of the target image The horizontal axis of .

[0034] According to the feature points of the target image The feature points corresponding to the previous frame of the target image The offset Determine the Y offset of the feature points in the Y direction column by column The details are as follows: In the formula, Indicates the Y offset of the feature point with row number i and column number j on the target image. Indicates the feature point with row number i and column number j on the target image The vertical coordinate of Indicates the feature point with row number i and column number j in the image of the previous frame of the target image The vertical coordinate of .

[0035] The sum of the X offsets / the sum of the Y offsets is determined according to the X offset / Y offset, and the row offset / column offset of the corresponding row / column is obtained by filtering. Specifically, the sum of the X offsets of the feature points is calculated row by row according to the X offset, and the number of feature points corresponding to each row is counted, as shown in the following formula: , In the formula, represents the sum of the X offsets of the i+1th row, Indicates the first X offset of the i+1th row, Indicates the last X offset of the i+1th row, represents the number of feature points in the i+1th row, Indicates the cols-th (last) X offset of the i+1-th row, where i represents the row number. This application starts numbering from 0.

[0036] It should be noted that in the above formula is only the sum of the X offsets of the feature points in the i+1th row, so The cols in The values ​​of j in are usually not equal. The value of depends on the order of a certain feature point among all the feature points in the row, and the value of j depends on the column number corresponding to the ordinate of a certain feature point in the row.

[0037] According to the Y offset, the sum of the Y offsets of the feature points is calculated column by column, and the number of feature points corresponding to each column is counted, as shown in the following formula: , In the formula, represents the sum of the Y offsets of the j+1th column, Indicates the first Y offset of the j+1th column, Indicates the last Y offset of the j+1th column, represents the number of feature points in the j+1th column, The j+1th column The last Y offset, j represents the column number, and this application starts numbering from 0.

[0038] It should be noted that in the above formula is only the sum of the Y offsets of the feature points in the j+1th column, so The row and The value of i in is usually not equal. The value of row depends on the order of a feature point among all the feature points in the column, and the value of i depends on the row number corresponding to the horizontal coordinate of a feature point in the row.

[0039] The filtering process here exemplarily uses moving mean filtering. The sum of the X offset and the sum of the Y offset are respectively subjected to moving mean filtering with a moving window size of a to obtain the row offset and the column offset, as follows: , In the formula, Indicates the row offset of the i+1th row, represents the sum of the X offsets of the i+1th row, represents the number of feature points in the i+1th row, Indicates the column offset of the j+1th column, represents the sum of the Y offsets of the j+1th column, represents the number of feature points in the j+1th column, represents the size of the moving window of the moving mean filter, i and j represent the row and column numbers of the feature points respectively, and this application starts numbering sequentially from 0.

[0040] The size of the moving window a is an odd number rounded up to the minimum resolution of the detectable target in the target image. For example, the minimum resolution of the detectable target is , then the size of the moving window a is 11. And here the values ​​of i and j should be greater than a, otherwise the feature point pairs and their feature point coordinates in the feature point matching results are re-determined.

[0041] The compensation offset is determined according to the row offset and the column offset. Specifically, the compensation offset of the feature point is finally calculated according to the row offset and the column offset, as follows: In the formula, Indicates the compensation offset of the feature point with row number i and column number j, Indicates the X offset of the feature point with row number i and column number j. Indicates the row offset of the i+1th row, Indicates the Y offset of the feature point with row number i and column number j. Indicates the column offset of the j+1th column.

[0042] Step 204: Determine whether the shooting roll angle is within the first angle range. For example, the first angle range is set to 5°. When the shooting roll angle is within When the shooting roll angle is not within the range, step 206 is executed. Within the range, that is, the shooting roll angle is within or When the range is reached, step 205 is executed.

[0043] Step 205: The offset compensation method is to first perform perspective transformation offset compensation, and then perform moving mean offset compensation. Specifically, the moving mean offset compensation mainly solves the problem that when the drone rolls, the perspective transformation offset compensation cannot solve the problem of random reduction direction of the offset. For example, when the angle between the pod camera equipped on the drone and the drone is 35°, due to the change in the roll angle of the drone, the direction of reduction of the column vector of the feature point in the perspective transformation is not certain. It is no longer reduced from the bottom to the top, and the row vector may be reduced from the left to the right. The moving mean offset compensation can adjust the offset change caused by the roll angle of the drone in real time, and can be applicable to fixed-wing drones.

[0044] Step 206: The offset compensation method is perspective transformation offset compensation. In the embodiment of the present application, the steps of the perspective transformation offset compensation method are as follows: Figure 3 As shown, it includes steps 301 to 305, which are specifically as follows.

[0045] Step 301: Construct a compensation table according to the imaging law of the target image. The compensation table includes an X-direction compensation table and a Y-direction compensation table. In the embodiment of the present application, the X-direction compensation table and the Y-direction compensation table are constructed according to the pixel imaging law of the target image. The size of the target image is (Indicates width height), the size of the compensation table is The height H of the compensation table is consistent with the height H of the target image, and the width of the compensation table is ,as follows: , where Indicates the width of the compensation table, represents the angle between the drone and the pod camera when shooting the target image, and W and H represent the width and height of the target image, respectively. For example, if the width and height of the target image here are , is 35°, then Width of compensation table when rounding down is 20.

[0046] Those skilled in the art should understand that the initial values ​​of each row vector and column vector in the compensation table constructed here are the X offset and Y offset of the corresponding feature point, respectively. The calculation results related to trigonometric functions in this application can be uniformly rounded up or down.

[0047] Step 302: Set the value of the row vector in each row according to the height of the X-direction compensation table. In the embodiment of the present application, the value of the middle position row vector of each row in the X-direction compensation table is set to half the height of the X-direction compensation table. Taking the middle position row vector as a reference, the values ​​of the row vectors within the preset range at both ends are sequentially set with the first difference as the tolerance, and the values ​​of the row vectors outside the preset range are set to the values ​​of the row vectors within the preset range closest to it. The preset range is determined according to the width of the X-direction compensation table, the row number of the middle position row vector, and the angle between the drone and the pod camera when shooting the target image.

[0048] The calculation method of the preset range is as follows: In the formula, Indicates the preset range, Indicates the width of the compensation table, Represents the height of the i+1th row, where The value of can be understood as the row number of the i+1th row in the X-direction compensation table. Indicates the angle between the drone and the pod camera when shooting the target image. The value of is exemplarily rounded down.

[0049] It should be noted that if the number of columns in the X-direction compensation table is an even number, one of the two row vectors at the middle position is selected as the middle position row vector.

[0050] Exemplarily, the height of the X-direction compensation table is 11 (corresponding to 11 rows in the compensation table, with row numbers sequentially numbered from 0 to 10), and the width is 20 (corresponding to 20 columns in the compensation table, with column numbers sequentially numbered from 0 to 19), then the value of the row vector in the middle position, that is, the row vector in the 11th column (column number 10), is set to 5.5.

[0051] Taking the middle position as the reference, 1 is the first tolerance, and the preset range n calculated by the above calculation method is as follows: n in the first row is 10, n in the second row is 10, n in the third row is 9, n in the fourth row is 8, n in the fifth row is 7, n in the sixth row is 6, n in the seventh row is 6, n in the eighth row is 5, n in the ninth row is 4, n in the tenth row is 3, and n in the eleventh row is 2.

[0052] According to the above data, the X-direction compensation table is shown in Table 1 below. The values ​​of the 10 row vectors on the left side of the middle position row vector in the 1st row and 11th column of the X-direction compensation table are set to 6.5, 7.5, 8.5, 9.5, ..., 15.5, and the values ​​of the 9 row vectors on the right side are set to 6.5, 7.5, 8.5, 9.5, ..., 14.5. The values ​​of the 10 row vectors on the left side of the middle position row vector in the 2nd row and 11th column of the X-direction compensation table are set to 6.5, 7.5, 8.5, 9.5, ..., 15.5, and the values ​​of the 9 row vectors on the right side are set to 6.5, 7.5, 8.5, 9.5, ..., 14.5. The values ​​of the 10 row vectors on the left side of the middle position row vector in the 3rd row and 11th column of the X-direction compensation table are set to 6.5, 7.5, 8.5, 9.5, ..., 14.5, 14.5, and the values ​​of the 9 row vectors on the right side are set to 6.5, 7.5, 8.5, 9.5, ..., 14.5. Similarly, the values ​​of the 10 row vectors on the left side of the middle position row vector in the 11th row and 11th column of the X-direction compensation table are set to 6.5, 7.5, 7.5, ..., 7.5, and the values ​​of the 9 row vectors on the right side are set to 6.5, 7.5, 7.5, ..., 7.5.

[0053] Table 1 X-direction compensation table

[0054] Step 303: Fill the values ​​of the column vectors in each column according to the data distribution of the offset in the Y-direction compensation table. In an embodiment of the present application, if the offset of the target image decreases from bottom to top, the value of the top column vector in the Y-direction compensation table is set to half the height of the Y-direction compensation table, and the values ​​of the remaining column vectors are set arithmetic differences from top to bottom with the second difference as the tolerance. If the offset of the target image increases from bottom to top, the value of the bottom column vector in the Y-direction compensation table is set to half the height of the Y-direction compensation table, and the values ​​of the remaining column vectors are set arithmetic differences from bottom to top with the second difference as the tolerance. Among them, the second difference is determined according to the angle between the drone and the pod camera when shooting the target image. If a negative value appears when setting the value of the column vector arithmetic differences, it is set to 0.

[0055] In the embodiment of the present application, the calculation formula of the second difference is as follows: . In the formula, b represents the second difference, Indicates the angle between the drone and the pod camera when capturing the target image.

[0056] For example, Take 35°, then the second difference b is 0.6. As shown in Table 2, the height of the Y-direction compensation table is 11 (corresponding to 11 rows in the compensation table, with row numbers numbered from 0 to 10), and the width is 20 (corresponding to 20 columns in the compensation table, with column numbers numbered from 0 to 19). If the data distribution of the offset in the Y-direction compensation table constructed according to the target image decreases from bottom to top, the value of the top column vector (i.e., the column vector in the first row, with row number 0) is half the height, 5.5, and the values ​​of the remaining column vectors decrease by 0.6 from top to bottom and are set to: 4.9, 4.3, 3.7, ... 0.1, 0.

[0057] Table 2 Y direction compensation table

[0058] Step 304: Determine the perspective transformation matrix of the target image based on the values ​​of the row vector and the column vector, perform perspective transformation on the compensation table through the perspective transformation matrix, and normalize it to obtain the perspective transformation compensation table. Specifically, the perspective transformation matrix is ​​calculated based on the four corner points of the target image and the four corner points after perspective transformation of the compensation table. According to the above Table 1 and Table 2, the corner points after perspective transformation are as follows: Figure 8 The calculation method of the number of columns of the corner points P3 and P4 after the perspective transformation of the compensation table is: the number of the center column of the compensation table (the 11th column, the column number is 10, so 10 is taken here) plus or minus the value of n in the last row (from Table 1 above, it can be seen that the value of n in the 11th row is 2).

[0059] The perspective transformation matrix can be calculated by substituting the feature point coordinates and the coordinates after perspective transformation of the compensation table (here, the corner points are used as feature points for example) into the linear least squares method or the direct linear transformation (DLT) algorithm to obtain a The perspective transformation matrix.

[0060] In the embodiment of the present application, the specific method of performing perspective transformation on the compensation table through the perspective transformation matrix is ​​as follows: , In the formula, It indicates the result of perspective transformation of the X-direction compensation table. represents the result of the Y direction compensation table after perspective transformation, x represents the X direction compensation table, y represents the Y direction compensation table, i and j represent the row number and column number corresponding to the feature point of perspective transformation in the perspective table, , , , , , , , and Represents an element in the perspective transformation matrix.

[0061] Then normalize the compensation table after perspective transformation to Interval, get the perspective transformation compensation table.

[0062] Step 305: Determine the compensation coefficient of the corresponding feature point in the perspective transformation compensation table according to the feature point coordinates of the feature point, and determine the compensation offset of the feature point through the compensation coefficient. Specifically, find the corresponding compensation offset coefficient in the perspective transformation compensation table according to the feature point coordinates of the feature point, and calculate the compensation offset of the feature point based on the compensation offset coefficient.

[0063] Step 207: Determine the compensation offset based on the determined offset compensation method. Specifically, according to the above navigation state and the shooting roll angle, perspective transformation offset compensation and / or moving mean offset compensation are used as the offset compensation method to finally obtain the compensation offset.

[0064] Step 105: Cluster the compensation offsets, and divide the targets into static targets and dynamic targets based on the clustering results. In the embodiment of the present application, the feature points are divided into static feature points and dynamic feature points based on the compensation offsets. Dynamic feature points are points and noise points on objects in motion. Static feature points are points and background points on objects in a stationary state, and the number of static feature points is greater than that of dynamic feature points. In practical applications, there may be some mismatched feature points, so the feature points also include a small number of noise points. Before dividing the static feature points into dynamic feature points, the noise points are first removed to ensure the accuracy of the final occupier judgment. Fig.10 As shown in the figure, the green dots are the static feature points of the division, and the red crosses are the dynamic feature points of the division.

[0065] Specifically, Figure 5 As shown, the compensation offset of feature points is divided into five categories through clustering algorithm, namely, k1, k2, k3, k4 and k5. The number of compensation offsets of feature points in the above five categories decreases from k1 to k5. Class k1 represents the compensation offset class of static feature points, class k2 represents the compensation offset class or retention class of dynamic feature points (the specific category of compensation offset is determined after judgment), class k3 represents the compensation offset class or retention class of dynamic feature points (the specific category of compensation offset is determined after judgment), and classes k4 and k5 represent the noise classes at both ends of the compensation offset.

[0066] Here, a k-means (English full name: K-means clustering algorithm, Chinese: K-means clustering algorithm) clustering algorithm is exemplarily adopted.

[0067] First, the two classes k4 and k5 with the least number of compensation offsets of feature points among the above five classes are eliminated.

[0068] Secondly, determine whether the number of images in the image sequence set is less than 3 frames. If the number of images in the image sequence set is less than 3 frames, save the class k1 with the largest compensation offset of the feature points, and update the target image until the number of images in the image sequence set reaches 3 frames or more, and then re-cluster and classify the compensation offsets of the feature points of the updated target image. If the number of images in the image sequence set is greater than or equal to 3 frames, calculate the mean m of the compensation offsets in the k1 class of the first three frames. Merge the k1 class in the first two frames of images with the k1 class, k2 class, and k3 class of the current frame image (target image), respectively, and then calculate the mean m1, m2, and m3 of the compensation offsets in the k1 class, k2 class, and k3 class in the first three frames of images after merging. If m2 is closest to m, exchange the compensation offsets in the k1 class and the k2 class. If m3 is closest to m, exchange the compensation offsets in the k1 class and the k3 class. If m1 is closest to m, maintain the compensation offsets in the original classes.

[0069] Then, the mean values ​​of compensation offsets mk1, mk2 and mk3 in the image (target image) of the current frame are calculated respectively. If mk2 is closest to mk1, then the k1 class and the k2 class are merged as the final compensation offset class for static feature points, and the k3 class is the final compensation offset class for dynamic feature points. If mk3 is closest to mk1, then the k1 class and the k3 class are merged as the final compensation offset class for static feature points, and the k2 class is the final compensation offset class for dynamic feature points.

[0070] Finally, according to the compensation offset class of static feature points and the compensation offset class of dynamic feature points divided as above, the feature points are divided into static feature points and dynamic feature points.

[0071] The number of feature points n on the target object is counted, and the number of static feature points n1 is determined. Then the variance s and mean t of the compensation offset of all static feature points on the target image are calculated, and then the mean t1 of the compensation offset of the static feature points on the target object is calculated.

[0072] If the number of static feature points n1 on the target object is greater than half of the number of feature points n on the target object, and the difference between the mean value t1 of the compensation offset of the static feature points on the target object and the mean value t of the compensation offset of all static feature points on the target image (i.e. ) is less than the variance s of the compensation offsets of all static feature points on the target image, the target is preliminarily judged as a static target. Otherwise, the target is a dynamic target.

[0073] Step 106: Determine whether the static target is in the threat area. In the embodiment of the present application, the threat area refers to the area above the pipeline that endangers the safety of the pipeline or affects the operation and maintenance of the pipeline. If the static target is in the threat area, execute step 108; if the static target is not in the threat area, execute step 107.

[0074] Step 107: The static target object is not an occupied object. Specifically, Fig. 9 As shown, if the static target is not in the threat area, it is not an occupied object.

[0075] Step 108: Determine whether the preset number of iterations has been reached. Specifically, during the object detection process, there are a few discontinuous temporary objects (such as temporary parking) that are misdetected. If the temporary objects are directly determined as occupied objects, the accuracy of the occupied object detection will be affected. Therefore, a secondary judgment is performed on static objects to make the detection results more stable and accurate. If the preset number of iterations has been reached, step 110 is executed. If the preset number of iterations has not been reached, step 109 is executed.

[0076] Step 109: The image of the previous frame of the target image where the static target is located is used as the target image. In the embodiment of the present application, the static targets are divided into permanent static targets (such as buildings) and non-permanent static targets (such as cars, tankers, etc.). Therefore, the static targets are judged on multiple frames of continuous images. The image of the previous frame of the target image is used as the target image, and step 103 is executed again until the preset number of iterations is reached. Fig.11 The following are non-permanent static objects to be detected. Fig.12 Shown is a permanently static target being inspected.

[0077] Step 110: The static target is an occupied object. Specifically, by determining whether the static target is in the threat area, it is possible to determine whether the static target is an occupied object, and then by executing steps 103 to 108 for a preset number of iterations, it is possible to determine whether the static target is a permanent static target or a non-permanent static target, thereby eliminating the interference of temporary targets and improving the accuracy of the detection result.

[0078] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in this embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in sequence or in parallel according to the method shown in this embodiment or the accompanying drawings (for example, in a parallel processor or multi-threaded processing environment).

[0079] like Figure 6As shown, the embodiment of the present application further provides an occupied object detection device 600 based on characteristic offset. The device comprises: an extraction module 601, a detection module 602 and an iteration module 603, as follows.

[0080] The extraction module 601 is used to extract the target object on the target image from the image sequence set.

[0081] The detection module 602 is used to detect feature points in the image sequence set.

[0082] The iteration module 603 is used to iteratively execute the classification step until a preset number of iterations is reached, and then determine that the static target object is an occupied object. The classification step includes: matching the target image and the image of the previous frame in the image sequence set based on the feature points. Determine the offset of the target image based on the feature point matching result, and perform offset compensation on the offset to obtain a compensated offset. Cluster the compensated offset, and divide the target into a static target and a dynamic target according to the clustering result. Determine whether the static target is in the threat area. If the static target is not in the threat area, determine that the static target is not an occupied object. If the static target is in the threat area, use the image of the previous frame of the target image where the static target is located as the target image to perform the classification step.

[0083] Some modules in the apparatus described in the present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0084] The devices or modules described in the above application embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described in various modules according to their functions. When implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, the module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0085] The methods, devices or modules described in this application can be implemented in the form of computer-readable program codes. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program codes (such as software or firmware) that can be executed by the (micro)processor, logic gates, switches, application-specific integrated circuits (English: Application Specific Integrated Circuit; Abbreviation: ASIC), programmable logic controllers and embedded microcontrollers. Examples of controllers include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program codes, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included in it for implementing various functions can also be regarded as structures within the hardware component. Or even, the means for realizing various functions may be regarded as both a software module for realizing the method and a structure within a hardware component.

[0086] An embodiment of the present application further provides a device, comprising: a processor; a memory for storing processor executable instructions; when the processor executes the executable instructions, the method described in the embodiment of the present application is implemented.

[0087] The embodiments of the present application also provide a non-volatile computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed, the method described in the embodiments of the present application is implemented.

[0088] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist independently, or two or more modules may be integrated into one module.

[0089] The above storage media include but are not limited to random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD) or memory card. The memory can be used to store computer program instructions.

[0090] It can be seen from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of the present application can be essentially or partly reflected in the prior art in the form of a software product, or it can be reflected in the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0091] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0092] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A method for detecting an occupied object based on a characteristic offset, characterized in that: include: Extracting the target object on the target image from the image sequence set; Detecting feature points in the image sequence set; Iteratively executing the classification step until a preset number of iterations is reached, then determining that the static target object is an occupied object; The classification step comprises: Based on the feature points, feature point matching is performed on the target image and the image of the previous frame in the image sequence set; Determining the offset of the target image based on the feature point matching result, and performing offset compensation on the offset to obtain a compensated offset; Clustering the compensation offsets, and dividing the targets into static targets and dynamic targets according to the clustering results; Determining whether the static target is in a threat area, and if the static target is not in the threat area, determining that the static target is not an occupied object; If the static target is within the threat area, the classification step is performed using the image of the previous frame of the target image where the static target is located as the target image.

2. The method according to claim 1, characterized in that: The performing offset compensation on the offset to obtain a compensated offset includes: Acquire the navigation state and shooting roll angle of the drone when shooting the target image; An offset compensation method is selected according to the navigation state and the shooting roll angle to determine the compensation offset; wherein the offset compensation method includes perspective transformation offset compensation and / or moving mean offset compensation.

3. The method according to claim 2, characterized in that The method of selecting an offset compensation method according to the navigation state and the shooting roll angle comprises: When the navigation state of the drone is a straight-line navigation state, and the shooting roll angle is within a first angle range, the offset compensation method is the perspective transformation offset compensation; When the navigation state of the drone is a straight-line navigation state, and the shooting roll angle exceeds the first angle range, the offset compensation method is to first perform the perspective transformation offset compensation, and then perform the moving mean offset compensation; When the navigation state of the UAV is a turning state, the offset compensation method is the moving mean offset compensation.

4. The method according to claim 2 or 3, characterized in that: The perspective transformation offset compensation includes: Constructing a compensation table according to the imaging rule of the target image; wherein the compensation table includes an X-direction compensation table and a Y-direction compensation table; Setting the value of the row vector in each row according to the height of the X-direction compensation table; Fill the value of the column vector in each column according to the data distribution of the offset in the Y direction compensation table; Determining the perspective transformation matrix of the target image based on the value of the row vector and the value of the column vector, performing perspective transformation on the compensation table through the perspective transformation matrix, and normalizing the compensation table to obtain a perspective transformation compensation table; The compensation coefficient of the corresponding feature point is determined in the perspective transformation compensation table according to the feature point coordinates of the feature point, and the compensation offset of the feature point is determined by the compensation coefficient.

5. The method according to claim 4, characterized in that The step of setting the value of the row vector in each row according to the height of the X-direction compensation table comprises: Setting the value of the middle position row vector of each row in the X-direction compensation table to half the height of the X-direction compensation table; Taking the middle position row vector as a reference, the values ​​of the row vectors within the preset ranges at both ends are sequentially set arithmetic differences with the first difference as the tolerance, and the value of the row vector outside the preset range is set to the value of the row vector within the preset range closest to it; wherein the preset range is determined according to the width of the X-direction compensation table, the row number of the middle position row vector, and the angle between the drone and the pod camera when shooting the target image.

6. The method according to claim 4, characterized in that Filling the value of the column vector in each column according to the data distribution of the offset in the Y direction compensation table includes: If the offset of the target image decreases from bottom to top, the value of the top column vector in the Y direction compensation table is set to half the height of the Y direction compensation table, and the values ​​of the remaining column vectors are set arithmetic differences from top to bottom with the second difference as the tolerance; If the offset of the target image increases from bottom to top, the value of the bottom column vector in the Y direction compensation table is set to half the height of the Y direction compensation table, and the values ​​of the remaining column vectors are set arithmetic difference from bottom to top with the second difference as the tolerance; wherein the second difference is determined according to the angle between the drone and the pod camera when shooting the target image; If a negative value occurs when setting the value of the column vector arithmetic progression, it is set to 0.

7. The method according to claim 2 or 3, characterized in that: The moving mean offset compensation includes: Determine the X offset / Y offset of the feature point in the X direction / Y direction row by row / column by row based on the offset of the target image; Determine the sum of the X offsets / the sum of the Y offsets according to the X offsets / the Y offsets, and perform filtering processing on the sum of the X offsets / the sum of the Y offsets to obtain the row offsets / column offsets of the corresponding rows / columns; The compensation offset is determined according to the row offset and the column offset.

8. An occupied object detection device based on characteristic offset, characterized in that: include: An extraction module, used for extracting the target object on the target image from the image sequence set; A detection module, used for detecting feature points in the image sequence set; An iteration module, used for iteratively executing the classification step until a preset number of iterations is reached, and then determining that the static target object is an occupied object; The classification step comprises: Based on the feature points, feature point matching is performed on the target image and the image of the previous frame in the image sequence set; Determining the offset of the target image based on the feature point matching result, and performing offset compensation on the offset to obtain a compensated offset; Clustering the compensation offsets, and dividing the targets into static targets and dynamic targets according to the clustering results; Determining whether the static target is in a threat area, and if the static target is not in the threat area, determining that the static target is not an occupied object; If the static target is within the threat area, the classification step is performed using the image of the previous frame of the target image where the static target is located as the target image.

9. A device for performing an occupied object detection method based on characteristic offset, characterized in that: include: processor; a memory for storing processor-executable instructions; When the processor executes the executable instructions, the method according to any one of claims 1 to 7 is implemented.

10. A non-volatile computer-readable storage medium, characterized in that: The device comprises a computer program or an instruction for storing the computer program or the instruction, which, when executed, enables the method according to any one of claims 1 to 7 to be implemented.