Sample enhancement method and apparatus

Abstract

The application discloses a sample enhancement method and device, relates to the technical field of model training, and comprises the following steps: determining the value range of a transformation parameter according to the scale transformation relationship of aerial images before and after perspective transformation, the geometric relationship between the shooting parameters of the aerial images, and the range of a target angle; determining a plurality of perspective transformation matrices by taking values in the value range of the transformation parameter; performing perspective transformation on an original image sample set by using the plurality of perspective transformation matrices to obtain a plurality of transformed images; updating the target label frame of the transformed images by using coordinate mapping to calibrate label information; and combining the labeled aerial images and the label information thereof with the transformed images and the calibrated label information of the transformed images to obtain an enhanced sample set. The application solves the problem that the existing training samples are difficult to adapt to actual flight postures and cannot overcome the performance decline of a target detection model caused by image distortion, improves the diversity of samples, and reduces the collection time required for collecting diverse samples.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model training, in particular to a sample enhancement method and device. BACKGROUND

[0002] Unmanned aerial vehicle (UAV) inspection refers to the use of UAVs equipped with camera devices to automatically or semi-automatically detect, monitor and collect data in target areas. UAV inspection is widely used in the fields of power inspection, agricultural monitoring and building surveying due to its high flexibility, wide coverage and high information collection efficiency.

[0003] In actual inspection, the flight attitude of the UAV is often difficult to maintain completely stable due to the influence of factors such as air flow and control accuracy, resulting in distortion of the collected aerial images. The flight attitude of the UAV is mainly controlled by the yaw angle, roll angle and pitch angle. The change of the yaw angle has little effect on the size and shape of the target in the aerial image, and can be corrected by image rotation. The change of the roll angle will cause morphological distortion of the aerial image, i.e. the attitude of the target in the aerial image is distorted. This distortion will cause the target detection model relying on the shape feature to produce misjudgment or missed detection. The change of the pitch angle will cause double distortion of the aerial image, i.e. the shape and scale of the target in the aerial image change simultaneously, which is significantly different from the real shape, resulting in misjudgment, missed detection or large size measurement error of the target detection model.

[0004] The training logic of the target detection model is to establish a database based on the standard view angle, fixed scale and regular shape of the target feature. Different pitch angles or roll angles will cause different degrees of distortion of the target in the aerial image, resulting in a large deviation of the actually collected image data from the "standard feature" in the training database, and thus a significant decrease in the detection accuracy, recognition accuracy and recall rate of the target detection model, and even the inability to complete the detection task. SUMMARY

[0005] The embodiments of the present application provide a sample enhancement method and device, which solve the problem that the existing training samples are difficult to adapt to the actual flight attitude and cannot overcome the performance degradation of the target detection model caused by image distortion.

[0006] In a first aspect, the embodiments of the present application provide a sample enhancement method, comprising: constructing an original image sample set based on annotated aerial images; determining a value range of a transformation parameter according to a scale transformation relationship between aerial images before and after perspective transformation, a geometric relationship between shooting parameters of the aerial images, and a range of a target angle; determining a plurality of perspective transformation matrices by taking values in the value range of the transformation parameter; performing perspective transformation on the aerial images in the original image sample set by using the plurality of perspective transformation matrices to obtain a plurality of transformed images; updating a target annotation box in the transformed images by using coordinate mapping to calibrate annotation information; and combining the annotated aerial images and their annotation information with the transformed images and their calibrated annotation information to obtain an enhanced sample set.

[0007] In combination with the first aspect, in a possible implementation manner, before the original image sample set is constructed based on the annotated aerial images, the method further comprises: performing cleaning processing on the aerial images to remove noise thereon; and / or performing defogging enhancement on the aerial images to improve their clarity; and / or standardizing the shooting parameters of the aerial images; and / or performing geometric correction and segmentation on the aerial images.

[0008] In combination with the first aspect, in a possible implementation manner, the method for determining the scale transformation relationship between the aerial images before and after perspective transformation comprises: determining a plurality of original feature points in the aerial images, and obtaining a plurality of target feature points after performing perspective transformation on the aerial images; and determining the scale transformation relationship between the aerial images before and after perspective transformation according to changes in distances between different original feature points and distances between corresponding target feature points.

[0009] In combination with the first aspect, in a possible implementation manner, the method for determining the geometric relationship between the shooting parameters of the aerial images comprises: the shooting parameters comprise a field of view angle, a field of view region, a height, and an attitude angle of an aerial device; and determining the geometric relationship between the pitch angle / roll angle and the field of view according to a relationship between a distance of the aerial device from the field of view region and the field of view angle.

[0010] In combination with the first aspect, in a possible implementation manner, the method for determining the value range of the transformation parameter comprises: determining the value range of the transformation parameter according to the range of the target angle, the geometric relationship, and the scale transformation relationship.

[0011] In combination with the first aspect, in a possible implementation manner, the method for determining the plurality of perspective transformation matrices by taking values in the value range of the transformation parameter comprises: determining coordinates of original feature points and target feature points according to the transformation parameter and the scale transformation relationship; and determining a perspective transformation matrix corresponding to the transformation parameter according to the coordinates of the original feature points and the target feature points.

[0012] With reference to the first aspect, in a possible implementation manner, the updating the target bounding box in the transformed image by using the coordinate mapping comprises: determining coordinates of a new target bounding box in the transformed image according to corner point coordinates of the original target bounding box and the perspective transformation matrix; determining the new target bounding box that is cropped at an edge based on an interaction relationship between the original target bounding box and the new target bounding box; deleting an unqualified target bounding box according to an area and an aspect ratio of the original target bounding box and the new target bounding box; and updating the target bounding box and the labeling information of the transformed image according to the remaining new target bounding boxes and class information of the new target bounding boxes.

[0013] With reference to the first aspect, in a possible implementation manner, the determining the new target bounding box that is cropped at an edge based on an interaction relationship between the original target bounding box and the new target bounding box comprises: if three corner points of the original target bounding box are cropped in the transformed image, and an original center point of the original target bounding box is not on a cropping line, the original target bounding box is deleted; if three corner points of the original target bounding box are cropped in the transformed image, and the original center point of the original target bounding box is on the cropping line, a new target bounding box is constructed with the original center point as a corner point; if two corner points of the original target bounding box are cropped in the transformed image, the remaining original target bounding box is taken as the new target bounding box; and if one corner point of the original target bounding box is cropped in the transformed image, a new target bounding box is constructed with an intersection point of a diagonal line of the original target bounding box and the cropping line as a corner point.

[0014] With reference to the first aspect, in a possible implementation manner, the deleting the unqualified target bounding box according to an area and an aspect ratio of the original target bounding box and the new target bounding box comprises: if the area of the new target bounding box reaches half of the area of the original target bounding box, the aspect ratio is compared; if a ratio of the aspect ratio of the new target bounding box to the aspect ratio of the original target bounding box is within a preset range, the new target bounding box is retained; if the ratio of the aspect ratio of the new target bounding box to the aspect ratio of the original target bounding box is not within the preset range, the new target bounding box is deleted; and if the area of the new target bounding box does not reach half of the area of the original target bounding box, the new target bounding box is deleted.

[0015] In a second aspect, the embodiments of the present application provide a sample enhancement device, comprising: a construction module configured to construct an original image sample set based on labeled aerial images; a transformation parameter module configured to determine a value range of a transformation parameter according to a scale transformation relationship between aerial images before and after perspective transformation, a geometric relationship between shooting parameters of the aerial images, and a range of target angles; a perspective transformation matrix module configured to take values in the value range of the transformation parameter to determine a plurality of perspective transformation matrices; a perspective transformation module configured to perform perspective transformation on aerial images in the original image sample set by using the plurality of perspective transformation matrices to obtain a plurality of transformed images; a calibration module configured to update target label boxes in the transformed images by using coordinate mapping to calibrate label information; and a combination module configured to combine the labeled aerial images and label information thereof with the transformed images and calibrated label information thereof to obtain an enhanced sample set.

[0016] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0017] By determining the value range of the transformation parameter, the embodiments of the present application can determine a plurality of different perspective transformation matrices, and then perform multi-angle perspective transformation on the original images to enrich the diversity of the samples. By calibrating the label information of the transformed images after perspective transformation, the embodiments of the present application can ensure the consistency of the image spatial information and the label information, and improve the reliability of model training. The embodiments of the present application effectively solve the problem that the existing training samples are difficult to adapt to actual flight postures, and cannot overcome the performance degradation of the target detection model caused by image distortion, and further improve the diversity of the samples, so that the target detection model can better detect target features with different distortions, and also reduce the collection time required for collecting diverse samples, and improve the generalization performance of the target detection model to the target environment. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0019] Figure 1 A flowchart of a sample enhancement method provided by the embodiments of the present application;

[0020] Figure 2 An example diagram of forward perspective transformation provided by the embodiments of the present application;

[0021] Figure 3 An example diagram of negative perspective transformation provided by the embodiments of the present application;

[0022] Figure 4 An example diagram of a relationship between a shooting parameter of a shooting device and a ground field of view is provided for an embodiment of the present application.

[0023] Figure 5 An example diagram of a flight image is provided for an embodiment of the present application.

[0024] Figure 6 An example diagram of a perspective transformation of the target object is provided for an embodiment of the present application. Figure 5 An example diagram of a perspective transformation of the target object is provided for an embodiment of the present application.

[0025] Figure 7 An example diagram of a perspective transformation of the target object is provided for an embodiment of the present application.

[0026] Figure 8 An example diagram of a perspective transformation of the target object is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of, rather than all of, the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0028] The following descriptions of some technologies related to the embodiments of the present application are provided to help understanding, which should be considered as merely exemplary. Therefore, a person of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, some descriptions of well-known functions and structures are omitted in the following description for clarity and brevity.

[0029] Figure 1 FIG. 1 is a flowchart of a sample enhancement method provided by an embodiment of the present application, including steps 101 to 106. In this embodiment, the sample enhancement method includes the following steps. Figure 1 The execution order shown in the embodiments of the present application is only one execution order, and the sample enhancement method can be executed in other execution orders. Figure 1 The steps shown in the embodiments of the present application can be executed in parallel or in reverse.

[0030] Step 101: Construct an original image sample set based on the labeled flight image. In the embodiments of the present application, in order to make the trained target detection model better adapt to the complex scene of the flight process, the unmanned aerial vehicle can adaptively adjust the flight height, sensor parameters and shooting angle to optimize the data resolution and coverage. Moreover, the collected flight images should cover the patrol scenes under different seasons, different lighting conditions, different weather conditions and different shooting angles.

[0031] In the embodiments of the present application, in order to ensure the reliability of the sample data of the target detection model, the aerial image can also be preprocessed, including: cleaning the aerial image to remove noise thereon. And / or, performing defogging enhancement on the aerial image to improve its clarity. And / or, standardizing the shooting parameters of the aerial image. And / or, performing geometric correction and segmentation on the aerial image.

[0032] Specifically, the aerial image is cleaned to remove noise interference and sensor interference in the aerial image through a denoising algorithm, a filtering algorithm, etc. Or the aerial image is subjected to defogging enhancement processing to improve the clarity of the aerial image. The shooting parameters of the aerial image can also be standardized to unify the data format and dimension of different shooting devices and collection times. The aerial image can also be rotated, cropped, and segmented to extract key regions such as inspection targets and improve the quality of the aerial image and the accuracy of subsequent feature extraction.

[0033] The mainstream training method of the target detection model is supervised training, so the samples need to be labeled. By adding accurate target bounding boxes in the aerial image and labeling the types of targets in the target bounding boxes, the target detection model is guided to learn. In order to improve the accuracy of the labeling information, the first labeling generally adopts traditional manual labeling, and the labeling content includes manually selecting a target and marking the category of the target, and the target bounding box is required to be the minimum bounding rectangle of the target, and the categories of targets of the same category are uniform.

[0034] Finally, the aerial image after preprocessing and labeling is stored as an original image sample set.

[0035] Step 102: According to the scale transformation relationship of the aerial image before and after perspective transformation, the geometric relationship between the shooting parameters of the aerial image, and the range of the target angle, the value range of the transformation parameter is determined. In the embodiments of the present application, the determination method of the scale transformation relationship of the aerial image before and after perspective transformation includes: determining a plurality of original feature points in the aerial image, and obtaining a plurality of target feature points after perspective transformation of the aerial image. According to the change of the distance between different original feature points and the distance between the corresponding target feature points, the scale transformation relationship before and after the perspective transformation of the aerial image is determined.

[0036] In the embodiments of the present application, the determination method of the geometric relationship between the shooting parameters of the aerial image includes: the shooting parameters include the field of view angle, the field of view region, the height, and the attitude angle of the aerial device. According to the relationship between the distance of the aerial device from the field of view region and the field of view angle, the geometric relationship between the pitch angle / roll angle and the field of view is determined.

[0037] It should be noted that the application does not limit the determination manner of the range of the target angle, which can be determined based on the allowed shooting angle range or the commonly used angle range of the aerial photography device, or can be formulated by the user based on actual needs. Preferably, the preset range can be extended based on the shooting angle range of the original image sample set.

[0038] In the embodiment of the application, the value range of the transformation parameter is determined according to the range of the target angle, the geometric relationship and the scale transformation relationship.

[0039] Specifically, the perspective transformation aims to solve the distortion problem of the aerial photography image of the unmanned aerial vehicle caused by the change of the attitude angle (yaw angle, roll angle and pitch angle, especially the roll angle and the pitch angle), and the image correction is realized by accurately constructing the perspective transformation matrix, thereby improving the reliability of the inspection analysis. The core of the transformation parameter generation process is to solve the perspective transformation matrix of 3 3, which is calculated by establishing the geometric relationship of a plurality of (exemplarily four) original feature points and a plurality of (exemplarily four) target feature points.

[0040] Further, since the change of the pitch angle mainly affects the deformation of the aerial photography image in the horizontal direction, and the change of the roll angle mainly affects the deformation of the aerial photography image in the vertical direction, the horizontal direction can be realized by rotating the aerial photography image by 90°, and the deformation after rotation also becomes horizontal. In addition, the aerial photography device (such as an optical pod) is bound with the unmanned aerial vehicle, and the pitch angle reflects the included angle between the aerial photography device and the horizontal line, which is generally negative downward. If the pitch angle is less than 30°, the target feature viewed by the front view of the aerial photography device is not enough to express the target attribute. If the pitch angle is equal to 90°, it belongs to the front view target, and the front view image is obtained. The change range of the pitch angle of the unmanned aerial vehicle is generally about ±10°. The correction of the deformation caused by the roll angle is consistent with the correction of the deformation caused by the pitch angle, and the following embodiments will be described in detail taking the pitch angle as an example.

[0041] Specifically, the selection of the original feature points and the target feature points in the image perspective transformation should follow the three core principles of non-collinearity, clear correspondence, and feature stability to ensure that the perspective transformation matrix can accurately map the aerial image to the target view. Generally, the original feature points are selected from four non-collinear significant feature points in the aerial image, which preferentially satisfy the geometric distribution rule and anti-interference, and are usually selected from the corner points, contour points, and reference object vertices. During the flight of the unmanned aerial vehicle, the aerial images are composed of continuous image frames, and there is a strong correlation between the contents of the aerial images. The scale and viewing angle of the target in the aerial image continuously change. For example: scale difference: at a long distance, the imaging size of the target is small and the details are blurred, and at a short distance, the target details are rich but the field of view is narrow; attitude deflection: under an inclined viewing angle, rectangular targets such as vehicles and building walls may exhibit trapezoidal or irregular quadrilateral distortion, and vehicles may exhibit different attitudes facing the lens. Moreover, the background image in which the target is located is also an influencing factor for target detection, and the scale and attitude of the target may also change, so the original feature points of the perspective transformation can be selected from the entire image, i.e., the top-left, top-right, bottom-left, and bottom-right corner points of the aerial image.

[0042] However, due to the three-dimensional change of the flight angle, the features are no longer limited to the unique change of a certain direction axis, and are reflected as a comprehensive change of the original feature points to the diagonal direction, which can be positive or negative. Among them, the positive change can stretch the compressed size of the long-distance target to the normal proportion, such as correcting a small-size vehicle at a long distance to a normal-size orthographic image at a short distance; can correct the converging state (such as the road edge) to a parallel state; can enhance the target features, although the resolution of the original image of the long-distance target is low, but after transformation, the local clarity can be improved through interpolation algorithm, providing a more regular image basis for subsequent analysis. The negative change can convert irregular perspective distortion, such as trapezoidal and diamond, into regular geometric shapes (such as rectangle and square). For example, a building photographed at a short distance presents a trapezoidal shape due to the downward viewing angle, and is restored to a horizontal orthographic rectangle after perspective transformation; the relative proportion of each part of the target can be restored, for example, the head part of a vehicle is stretched and the tail part is compressed due to the side viewing angle at a short distance, and after perspective transformation, it can be corrected to an orthographic view, making the length and width proportions of the vehicle body consistent with the actual physical size; the target detail features can be enhanced, and the orthographic aerial image after perspective transformation can avoid the occlusion or distortion of details caused by the viewing angle tilt, which helps to improve the target recognition accuracy.

[0043] Exemplarily, as Figure 2 and Figure 3The figure shows the positive perspective transformation and the negative perspective transformation. The green box represents the original aerial image, A0, B0, C0, D0 are four original feature points (corner points) in the aerial image, w0, h0 are the width and height of the aerial image, w, h are the width and height of the aerial image after perspective transformation, that is, the black frame trapezoidal part in the figure, whose coordinates are A1, B1, C1, D1. Four original feature points A0, B0, C0, D0 in the aerial image are transformed into four target feature points A1, B1, C1, D1. A, B, C, D are the image area range cropped from the aerial image after perspective transformation, that is, the red box part in the figure, d represents the horizontal transformation parameter of the original feature point after perspective transformation, and y represents the vertical transformation parameter of the original feature point after perspective transformation.

[0044] Further, the coordinates of the original feature points and the target feature points are represented with the point where D0 or D1 is located as the origin. The coordinates of the four original feature points are represented as D0(0, 0), C0(w0, 0), B0(w0, h0), and A0(0, h0) with D0 as the origin.

[0045] Figure 2 The coordinates of the four target feature points after positive perspective transformation can be represented as D1(0, 0), C1(w0, 0), B1(w0+d, h), and A1(-d, h).

[0046] According to the changes in the distances between different original feature points and the corresponding target feature points, the scale transformation relationship before and after the perspective transformation of the aerial image is determined. That is, according to the changes in the width w0 / height h0 of the aerial image and the corresponding width w / height h after perspective transformation, the scale transformation relationship before and after the perspective transformation of the aerial image is determined as follows:

[0047] ,

[0048] ,

[0049] .

[0050] Figure 3 The coordinates of the four target feature points after negative perspective transformation can be represented as D1(0, 0), C1(w0, 0), B1(w0-d, h0-y), and A1(d, h0-y).

[0051] The scale transformation relationship before and after the perspective transformation of the aerial image in the negative perspective transformation is as follows:

[0052] ,

[0053] ,

[0054] .

[0055] Furthermore, to adapt to the perspective transformation of aerial images caused by changes in pitch angle during actual drone inspections, the range of values ​​for the transformation parameters is determined. For example... Figure 4 As shown in the figure, point P is the current position of the aerial photography device, PL represents the flight direction of the drone, point Q is the ground projection corresponding to point P, point O is the intersection of the imaging center line of the aerial photography device and the ground, points A, B, C, and D are the intersections of the horizontal and vertical fields of view of the camera with the ground under the current attitude flight view, that is, the corner points of the ground's field of view area, points E and F are the intersections of the drone's flight direction with the field of view area, PF represents the distance of the current position of the aerial photography device from the farthest vertical field of view area, PE represents the distance of the current position of the aerial photography device from the nearest vertical field of view area, PQ represents the height of the aerial photography device above the ground, ∠LPO represents the pitch angle of the aerial photography device, ∠FPE represents the vertical field of view angle of the aerial photography device, and ∠BPC represents the horizontal field of view angle of the aerial photography device.

[0056] If the pitch angle of the aerial photography device at its current position is θ, then the following relationship can be derived:

[0057] ,

[0058] ,

[0059] ,

[0060] ,

[0061] This allows us to determine the geometric relationship between the pitch angle and the field of view:

[0062] .

[0063] For example, if the pitch angle θ = 90°, the vertical field of view ∠FPE of the aerial photography device is 60°, the horizontal field of view ∠BPC of the aerial photography device is 60°, and the field of view width of the front view image is BC = 1080px (pixels), based on the proportional relationship, the field of view width of the front view image can be calculated as follows: when the pitch angle θ = 30°, the field of view width of the front view image is BC = 5341px; when the pitch angle θ = 45°, the field of view width of the front view image is BC = 3612px; and when the pitch angle θ = 60°, the field of view width of the front view image is BC = 1869px.

[0064] Since the pitch angle θ is usually about 45° in actual flight, the lateral resolution of the aerial image is assumed to be 1080px. According to the scaling relationship, when the pitch angle θ = 30°, the lateral resolution of the aerial image is 1596px; when the pitch angle θ = 60°, the lateral resolution of the aerial image is 558px; and when the pitch angle θ = 90°, the lateral resolution of the aerial image is 322px.

[0065] It should be noted that when the flight height of the unmanned aerial vehicle changes, the projection of the inherent geometry of the ground object in the aerial image is relatively stable, and the changes are mainly the changes in spatial resolution and scene coverage caused by the change in the field of view of the imaging system. In the typical aerial height range, which is much higher than the size of the ground object, the height change mainly causes the scaling between the imaging system and the scene, rather than the perspective distortion. The contour and relative proportion of the ground object in the aerial image are basically determined by its actual geometry and orientation, and the height increase or decrease mainly causes the ground object to be enlarged or reduced in the aerial image as a whole, and the projection geometry of the shape features (such as the rectangular contour of the building, the linear structure of the road, and the length-width ratio of the vehicle) remains relatively consistent. In addition, the above calculation process is only used to calculate a range of values of the transformation parameter (d), and the value in the range is used to calculate the perspective transformation matrix, so the influence of the aerial height on the transformation parameter in different flight processes can be ignored.

[0066] According to the above scaling relationship, the transformation parameter belongs to half the number of pixels in the lateral direction in which the resolution of the aerial image changes, that is: .

[0067] Therefore, the negative perspective change of the pitch angle of 30° is calculated to be d = -258, the positive perspective change of the pitch angle of 60° is calculated to be d = 261, and the positive perspective change of the pitch angle of 90° is calculated to be d = 379. Therefore, it can be known that the angle range of the pitch angle in actual flight is between [-30°, -90°], and therefore the change range of the above transformation parameter d is [-258, 379].

[0068] Step 103: Taking a value in the range of the transformation parameter to determine a plurality of perspective transformation matrices. In the embodiment of the present application, the coordinates of the original feature points and the target feature points are determined according to the transformation parameter and the scaling relationship. According to the coordinates of the original feature points and the target feature points, the perspective transformation matrix corresponding to the transformation parameter is determined.

[0069] Specifically, the coordinates of the original feature points and the target feature points are determined according to the transformation parameter d and the scaling relationship, that is, Figure 2 or Figure 3The coordinates of the four groups of feature points A0, B0, C0, D0, A1, B1, C1, and D1.

[0070] For example, assuming that the coordinates of an original point on the aerial image are (x, y, z), the original point is subjected to perspective transformation by the perspective transformation matrix of 3 3 to obtain first coordinates (X, Y, Z). The first coordinates are normalized according to the value of Z, and the normalized first coordinates obtain homogeneous coordinates (X', Y', 1).

[0071] The following is the process of normalizing the first coordinates:

[0072] .

[0073] When the homogeneous coordinates , the point is the two-dimensional plane coordinates of the original point after perspective transformation, and the following relationship can be obtained:

[0074] ,

[0075] When the settlement is performed, let be expanded to obtain:

[0076] ,

[0077] In the formula, , , , , , , , , represent the 9 elements in the perspective transformation matrix of 3 3.

[0078] In the above expanded equation, there are a total of 8 unknowns, which can be solved by using the coordinates of the original feature points and the target feature points obtained above to solve the values of the elements , , , , , , , in the perspective transformation matrix, thereby obtaining the perspective transformation matrix.

[0079] It should be noted that when the values in the value range of the transformation parameters are taken, they can be randomly taken or user-specified, uniformly taken, etc. Preferably, the values are taken relatively dispersedly in the value range.

[0080] Exemplarily, the transformation parameter d is valued according to a uniform distribution, a plurality of groups of coordinates of the original feature points and the target feature points are determined by the above method, and then the values of the elements in different perspective transformation matrices are solved to obtain a plurality of perspective transformation matrices.

[0081] Step 104: Perspective transformation is performed on the aerial images in the original image sample set by using the plurality of perspective transformation matrices to obtain a plurality of transformed images. In the embodiment of the present application, the aerial images are subjected to perspective transformation by using the cv2.warpPerspective function (a function for performing perspective transformation on images). The cv2.warpPerspective function traverses each pixel in the aerial image, calculates the coordinates after perspective transformation according to the perspective transformation matrix, fills the pixel values by using an interpolation algorithm, generates a new image in which the distortion is eliminated, and then the new image is cropped to obtain a transformed image. As shown in the figure, Figure 5 is the original aerial image, Figure 6 is the Figure 5 transformed image after perspective transformation.

[0082] It should be noted that the number of aerial images subjected to perspective transformation under each perspective transformation matrix should be similar or equal, that is, the number of transformed images obtained after perspective transformation by each perspective transformation matrix should be similar or equal.

[0083] Step 105: The target bounding box in the transformed image is updated by using coordinate mapping to calibrate the labeling information. In the embodiment of the present application, the target bounding box in the transformed image is updated by using coordinate mapping, which includes: determining the coordinates of the new target bounding box in the transformed image according to the corner point coordinates of the original target bounding box and the perspective transformation matrix. Based on the interaction relationship between the original target bounding box and the new target bounding box, the new target bounding box that is cropped at the edge is determined. According to the area and the aspect ratio of the original target bounding box and the new target bounding box, the unqualified target bounding box is deleted. The target bounding box and the labeling information of the transformed image are updated according to the remaining new target bounding box and the class information thereof.

[0084] Specifically, the updating of the labeling information of the corresponding transformed image after perspective transformation is a key step to ensure that the labeling information is aligned with the space of the transformed image, and the core is to convert the original target bounding box in the aerial image into a new target bounding box in the transformed image by using a coordinate mapping algorithm, and to update the labeling information stored in the XML file synchronously. The aerial images taken during the flight of the unmanned aerial vehicle are continuous, and the position of the target in each aerial image is not fixed. After perspective transformation and image cropping, the target at the edge position is most affected, and the six situations shown in Figure 7 may occur. Figure 7In the figure, the dashed line represents the cutting line, the black and white parts together are the original target bounding box, the black part on one side of the cutting line represents the original target bounding box that is cut off, and the white part represents the newly constructed target bounding box. The point in the figure represents the original center point, that is, the intersection point of the diagonals of the original target bounding box.

[0085] For the above six cases, first, based on the interaction relationship between the original target bounding box and the new target bounding box, the new target bounding box is constructed, and then according to the area and aspect ratio of the original target bounding box and the new target bounding box, the unqualified target bounding box is deleted.

[0086] In the embodiment of the present application, based on the interaction relationship between the original target bounding box and the new target bounding box, the new target bounding box cut off at the edge is determined, including: if in the transformed image, three corner points of the original target bounding box are cut off, and the original center point of the original target bounding box is not on the cutting line, the original target bounding box is deleted. If in the transformed image, three corner points of the original target bounding box are cut off, and the original center point of the original target bounding box is on the cutting line, a new target bounding box is constructed with the original center point as a corner point. If in the transformed image, two corner points of the original target bounding box are cut off, the remaining original target bounding box is taken as the new target bounding box. If in the transformed image, one corner point of the original target bounding box is cut off, a new target bounding box is constructed with the intersection point of the diagonal of the original target bounding box and the cutting line as a corner point.

[0087] Exemplarily, when constructing the new target bounding box, the four edges of the new target bounding box should be parallel or coincident with the four edges of the original target bounding box.

[0088] In the embodiment of the present application, according to the area and aspect ratio of the original target bounding box and the new target bounding box, the unqualified target bounding box is deleted, including: if the area of the new target bounding box reaches half of the area of the original target bounding box, the aspect ratio is compared. If the ratio of the aspect ratio of the new target bounding box to the aspect ratio of the original target bounding box is within a preset range, the new target bounding box is retained. If the ratio of the aspect ratio of the new target bounding box to the aspect ratio of the original target bounding box is not within the preset range, the new target bounding box is deleted. If the area of the new target bounding box does not reach half of the area of the original target bounding box, the new target bounding box is deleted.

[0089] Exemplarily, the preset range is set based on the aspect ratio of the original target bounding box in the aerial image in the original image sample set, or can also be set based on the needs of different applications of the user.

[0090] Exemplarily, the preset range is [0.8, 1.2], that is, the ratio of the aspect ratio of the new target bounding box is [0.8, 1.2] and the product of the aspect ratio of the original target bounding box.

[0091] The four corner points of the original target annotation box are mapped by using the perspective transformation matrix to obtain the four corner points of the new target annotation box. In addition, the new target annotation box can be further optimized by the minimum circumscribed rectangle algorithm to ensure that it can completely wrap the target after perspective transformation.

[0092] Because of the randomness of the transformation parameters and the difference in the position of the target in the image, perspective transformation can cause some targets to partially exceed the image boundary, which needs to be deleted because such targets cannot fully express the characteristics of the target. Perspective transformation can cause some targets to appear distorted in aspect ratio, which needs to be deleted because such targets cannot accurately express the characteristics of the target. Perspective transformation can cause some target features to appear blurred, which needs to be deleted because such target features will reduce the robustness of the features.

[0093] Further, the new target annotation box can be judged for abnormalities, i.e., the gradient, size, and aspect ratio of the new target annotation box are determined, and it is determined whether they are within the effective range. If the gradient, area, and aspect ratio of the new target annotation box are within the effective range, the new target annotation box is retained. Otherwise, the new target annotation box and its annotation information are deleted. The annotation information of the remaining new target annotation box is counted, and is balanced according to the distribution of the target categories.

[0094] The above abnormality judgment can be implemented by a multi-scale detection head network. Specifically, the multi-scale detection head network is used to extract the features of the target, and the size range of the original target annotation box in the aerial image is calculated. If the new target annotation box is not within the range, the new target annotation box and its annotation information are deleted. The gradient and aspect ratio of the target annotation box can also be calculated using this method, or the annotation information of the original target annotation box is used to calculate, which is not described here.

[0095] According to the coordinate information of the updated new target annotation box and the target type (the target type of the original target annotation box is used), the annotation information is updated and stored as an XML format file to ensure the spatial consistency of the image data. The annotation information includes target annotation box coordinates, image size information, target set information, target category information, etc.

[0096] Step 106: Combine the labeled aerial image and its annotation information with the transformed image and its calibrated annotation information to obtain an enhanced sample set. In the embodiment of the present application, the transformed image and its corresponding annotation information after perspective transformation and annotation information calibration are combined with the original image sample set and its corresponding annotation information to construct an enhanced sample set.

[0097] Specifically, the construction process is divided into two cases:

[0098] The first case is the pitch angle change sample.

[0099] In the flight process of fixed-wing unmanned aerial vehicles, the flight path is generally pre-set. Except for the normal pitch angle changes in the take-off and landing stages, the pitch angle changes in the cruising stage are mainly caused by environmental factors such as atmospheric turbulence or wind shear. There are relatively many targets in the flight process, and the forms of the targets are also relatively many. In addition, since the pitch angle changes can be positive or negative during flight, perspective transformation not only needs randomness, but also needs to maintain the uniform distribution of perspective transformation. Therefore, in order to better match different forms under different pitch angles, the range of image perspective transformation parameters can be appropriately expanded, avoiding excessive positive or negative changes that cause angle imbalance, and avoiding target missed detection caused by insufficient pitch angle adaptation.

[0100] The second case: roll angle change sample.

[0101] In the flight process of fixed-wing unmanned aerial vehicles, the roll angle changes in the cruising stage are mainly caused by turbulence disturbance and turning needs. Since conventional oil and gas pipeline laying is straight-line dominant, in the inspection process, in order to avoid terrain, obstacles and other needs to fly turn, the target under the roll state appears relatively less, so the range of image perspective transformation parameters in this process can be appropriately reduced, avoiding false detection caused by excessive background transformation.

[0102] In addition, the balance of different categories of target data in the target detection model training is the key prerequisite to ensure the generalization ability and classification fairness of the target detection model. The core goal is to eliminate the model bias caused by the difference in the number of samples, and to ensure stable learning of both minority and majority categories of targets. Perspective transformation will increase or decrease the number of samples of different categories, so when constructing the enhanced sample set, the number of samples of different categories needs to be counted, and sample balancing needs to be performed according to the data balancing strategy, or the number or proportion of samples of different categories of targets in the enhanced sample set needs to be adjusted according to the needs, in order to meet the needs of different feature extraction or training.

[0103] The enhanced sample set is used to train the target detection model, such as a convolutional neural network model based on deep learning. The forward propagation calculates the prediction results, and the network parameters are updated through back propagation. During the training process, the performance is evaluated regularly on the validation set, and multiple local optimal models with the highest validation set accuracy are saved. Finally, the global optimal model is taken.

[0104] In order to improve the recognition accuracy and generalization ability of the target detection model, the training parameters need to be adjusted constantly, and the test set needs to be used to evaluate the trained target detection model offline. Considering that the risk needs to be warned in the actual inspection process, the detection rate is taken as an important indicator of the target detection model. The detection rate represents the proportion of targets whose position and category are both detected correctly among all targets.

[0105] To verify the effectiveness of the method of the present application, a basic sample set consisting of 8884 labeled training sets and 1480 verification sets selected from actual flight videos is used to train the yolov5x network model to obtain a basic model. Then, the method of the present application is used to perform perspective transformation on the basic sample set by random transformation parameters to form an enhanced sample set, and the yolov5x network model is trained to obtain an enhanced model. The target categories in the two sample sets include: “person”, “car / pickup”, “truck”, “tanker”, “engineering vehicle”, “bus”, “tricycle”, “build”, etc. The sample scenes are collected based on oil and gas pipeline inspection, and the scenes include forests, highways, parking lots, farmland, factory areas, etc.

[0106] Two test sets are constructed. Test set one: the data scene is roughly equivalent to the scene of the training sample (basic sample set, enhanced sample set), a total of 198 pictures. Test set two: the data scene has no relationship with the scene of the training sample, a total of 11 pictures. The test process is based on the majority of targets appearing in the actual flight process, mainly taking small targets “car” and large targets “truck” as statistical objects. The training and test results are shown in Table 1:

[0107] Table 1: Test result data table of the basic model and the enhanced model under different test sets

[0108]

[0109] In the table, Map@0.5 represents the average precision mean. Only when the intersection over union of the predicted box and the real box is greater than or equal to 0.5, the detection result is considered correct.

[0110] From the above detection results, it can be seen that the detection effect of the enhanced model of the present application is greatly improved compared with the basic model. Not only is the detection effect good for small targets and large targets, but also the generalization ability for unknown scenes is strong.

[0111] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps can be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one of the many step execution orders, and does not represent the only execution order. When the device or client product is executed in practice, it can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment) according to the method order shown in the embodiments or the drawings.

[0112] As Figure 8As shown, the embodiment of the present application further provides a sample enhancement device 800. The device comprises a construction module 801, a transformation parameter module 802, a perspective transformation matrix module 803, a perspective transformation module 804, a calibration module 805 and a combination module 806, and the details are as follows.

[0113] The construction module 801 is configured to construct an original image sample set based on the labeled aerial images.

[0114] The transformation parameter module 802 is configured to determine the value range of the transformation parameter according to the scale transformation relationship between the aerial images before and after the perspective transformation, the geometric relationship between the shooting parameters of the aerial images, and the range of the target angle.

[0115] The perspective transformation matrix module 803 is configured to take values in the value range of the transformation parameter to determine a plurality of perspective transformation matrices.

[0116] The perspective transformation module 804 is configured to perform perspective transformation on the aerial images in the original image sample set by using the plurality of perspective transformation matrices to obtain a plurality of transformed images.

[0117] The calibration module 805 is configured to update the target label frame in the transformed image by using the coordinate mapping to calibrate the label information.

[0118] The combination module 806 is configured to combine the labeled aerial images and the label information thereof with the transformed images and the calibrated label information thereof to obtain an enhanced sample set.

[0119] Some of the modules in the device described in the present application can be described in the general context of computer-executable instructions, such as program modules, which are executed by computers. Generally, program modules include routines, programs, objects, components, data structures, classes, and the like, which perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including storage devices.

[0120] The device or module described in the above embodiment can be implemented by a computer chip or entity, or by a product with certain functions. For the convenience of description, the above device is described as various modules with functions. In the implementation of the embodiment of the present application, the functions of the modules can be implemented in the same or multiple software and / or hardware. Of course, the modules implementing certain functions can also be combined to implement the functions.

[0121] The methods, apparatuses or modules described in the present application can be implemented in a computer readable program code in any appropriate manner, for example, the controller can take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (for example, software or firmware) executable by the (micro)processor, logic gates, switches, application specific integrated circuits (ASIC), programmable logic controllers and embedded microcontrollers, examples of the controller include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in pure computer readable program code, the same function can be achieved by logically programming the method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers. Therefore, such a controller can be considered as a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both a software module for implementing the method and a structure within the hardware component.

[0122] The embodiments of the present application also provide a device, which comprises: a processor; a memory for storing processor executable instructions; and the processor implements the method as described in the embodiments of the present application when executing the executable instructions.

[0123] The embodiments of the present application also provide a non-volatile computer readable storage medium, which stores a computer program or instructions, and when the computer program or instructions are executed, the method as described in the embodiments of the present application is implemented.

[0124] In addition, each functional module in each embodiment of the present application can be integrated in one processing module, or each module can exist independently, or two or more modules can be integrated in one module.

[0125] The storage medium described above includes but is not limited to random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD) or memory card. The memory can be used to store computer program instructions.

[0126] From the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary hardware. Based on such an understanding, the technical solutions of the present application can be embodied in the form of a software product or in the form of data migration. The computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0127] The various embodiments in the specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. The whole or part of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, small computers, large computers, distributed computing environments including any of the above systems or devices, etc.

[0128] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the present application.

Claims

1. A sample augmentation method, characterized in that, include: Construct an original image sample set based on labeled aerial images; Based on the scale transformation relationship of the aerial images before and after perspective transformation, the geometric relationship between the shooting parameters of the aerial images, and the range of the target angle, the value range of the transformation parameters is determined to adapt to the perspective transformation of aerial images caused by changes in pitch angle during actual inspection by the UAV; wherein, the method for determining the geometric relationship between the shooting parameters of the aerial images includes: determining the geometric relationship between pitch angle / roll angle and field of view based on the relationship between the distance of the aerial device from the field of view area and the field of view angle; The values ​​of the transformation parameters are taken within the range to determine multiple perspective transformation matrices; The aerial images in the original image sample set are subjected to perspective transformation using multiple perspective transformation matrices to obtain multiple transformed images; The target bounding boxes in the transformed image are updated using coordinate mapping to calibrate the annotation information; wherein, updating the target bounding boxes in the transformed image using coordinate mapping includes: if the three corner points of the original target bounding box are clipped in the transformed image, and the original center point of the original target bounding box is not on the clipping line, then the original target bounding box is deleted. The labeled aerial images and their labeling information are combined with the transformed images and their calibrated labeling information to obtain an enhanced sample set.

2. The method according to claim 1, characterized in that, Before constructing the original image sample set based on the labeled aerial images, the process also includes: The aerial images are cleaned to remove noise; and / or, Dehazing and enhancing aerial images to improve their clarity; and / or, Standardize the shooting parameters of aerial images; and / or, Perform geometric correction and segmentation on aerial images.

3. The method according to claim 1, characterized in that, The method for determining the scale transformation relationship of the aerial images before and after perspective transformation includes: Multiple original feature points are identified in the aerial image, and multiple target feature points are obtained after perspective transformation of the aerial image; The scale transformation relationship before and after perspective transformation of aerial images is determined by the change in the distance between different original feature points and the distance between the corresponding target feature points.

4. The method according to claim 1, characterized in that, The shooting parameters include the field of view, field of view area, altitude, and attitude angle of the aerial photography device.

5. The method according to claim 1, characterized in that, Determining the range of values ​​for the transformation parameters includes: The range of values ​​for the transformation parameters is determined based on the range of the target angle, the geometric relationship, and the scale transformation relationship.

6. The method according to claim 3, characterized in that, The process of taking values ​​within the range of the transformation parameters to determine multiple perspective transformation matrices includes: The coordinates of the original feature points and the target feature points are determined based on the relationship between the transformation parameters and the scale transformation. Based on the coordinates of the original feature points and the target feature points, determine the perspective transformation matrix corresponding to the transformation parameters.

7. The method according to claim 1, characterized in that, The step of updating the target bounding box in the transformed image using coordinate mapping includes: The coordinates of the new target bounding box in the transformed image are determined based on the corner coordinates of the original target bounding box and the perspective transformation matrix. Based on the interaction between the original target bounding box and the new target bounding box, determine the new target bounding box that is clipped at the edge; Based on the area and aspect ratio of the original and new target annotation boxes, delete the unqualified target annotation boxes; Update the target bounding boxes and annotation information of the transformed image based on the remaining new target bounding boxes and their category information.

8. The method according to claim 7, characterized in that, The step of determining the new target annotation box that is clipped at the edge based on the interaction between the original target annotation box and the new target annotation box includes: If, in the transformed image, the three corner points of the original target bounding box are cropped, and the original center point of the original target bounding box is on the cropping line, then a new target bounding box is constructed using the original center point as the corner point. If two corner points of the original target bounding box are cropped in the transformed image, the remaining original target bounding box is used as the new target bounding box. If a corner of the original target bounding box is cropped in the transformed image, a new target bounding box is constructed using the intersection of the diagonal of the original target bounding box and the cropping line as the corner point.

9. The method according to claim 7, characterized in that, The step of deleting unqualified target annotation boxes based on the area and aspect ratio of the original and new target annotation boxes includes: If the area of ​​the new target annotation box is half the area of ​​the original target annotation box, then the aspect ratio is compared. If the aspect ratio of the new target annotation box to the original target annotation box is within the preset range, the new target annotation box will be retained. If the aspect ratio of the new target annotation box is not within the preset range compared to the aspect ratio of the original target annotation box, then delete the new target annotation box; If the area of ​​the new target annotation box is less than half the area of ​​the original target annotation box, then the new target annotation box is deleted.

10. A sample enhancement apparatus for implementing the method according to any one of claims 1-9, characterized in that, include: The building module is used to construct a raw image sample set based on labeled aerial images; The transformation parameter module is used to determine the value range of the transformation parameters based on the scale transformation relationship of the aerial images before and after perspective transformation, the geometric relationship between the shooting parameters of the aerial images, and the range of the target angle, so as to adapt to the perspective transformation of aerial images caused by changes in pitch angle during actual inspection by the UAV; wherein, the method for determining the geometric relationship between the shooting parameters of the aerial images includes: determining the geometric relationship between the pitch angle / roll angle and the field of view based on the relationship between the distance of the aerial device from the field of view area and the field of view angle; The perspective transformation matrix module is used to select values ​​within the range of the transformation parameters in order to determine multiple perspective transformation matrices; The perspective transformation module is used to perform perspective transformation on the aerial images in the original image sample set using multiple perspective transformation matrices to obtain multiple transformed images; The calibration module is used to update the target bounding boxes in the transformed image using coordinate mapping to calibrate the annotation information; wherein, updating the target bounding boxes in the transformed image using coordinate mapping includes: if the three corner points of the original target bounding box are clipped in the transformed image, and the original center point of the original target bounding box is not on the clipping line, then the original target bounding box is deleted. The combination module is used to combine the labeled aerial images and their annotation information with the transformed images and their calibrated annotation information to obtain an enhanced sample set.

Images