Unmanned aerial vehicle automatic driving auxiliary identification method and related device thereof

Abstract

The invention discloses an unmanned aerial vehicle automatic driving auxiliary identification method and a related device thereof. The method comprises the following steps: acquiring an inspection image; selecting an initial image area of a target object in the first frame of inspection image, and extracting a target feature from the initial image area of the target object in the first frame of inspection image; and training a filter by taking the target feature of the target object in the first frame of inspection image as a training sample, and identifying the position of the target object in the next frame of inspection image through the filter. According to the method, a model of the target equipment does not need to be collected in advance, only the position of the target object needs to be determined in the first frame inspection image, the target object in the subsequent inspection image is recognized based on the features of the target object, and the method can be directly deployed at the edge.

Description

Technical Field

[0001] The present application relates to the field of target recognition technology, and in particular to a method for assisting in the recognition of autonomous driving of unmanned aerial vehicles and related devices. Background Art

[0002] In the existing technical route, for drone inspection of 10kV overhead lines, the manual mode can be used, that is, the drone is manually controlled to align the power equipment, focus at different angles and take photos; the automatic driving mode can also be used, that is, the planned driving route and shooting points are used to take photos of the power equipment. Both modes have relatively high requirements for the shooting target. In power inspections, the power equipment is often not aligned in the manual stage or the planning stage, resulting in a greatly reduced inspection efficiency. Especially in manual mode, after the drone is in place, the camera needs to be strictly aimed at the equipment, which greatly wastes battery life. In addition, the traditional drone automatic identification method usually adopts a feature matching method. This method requires the model of the target equipment to be collected in advance and matched with the image collected by the drone during the inspection process. This method requires the model of the target equipment to be collected in advance, which is difficult to deploy at the edge. Summary of the invention

[0003] The present application provides a method for assisting in identifying an autonomous driving drone and a related device thereof, which are used to improve the technical problem that the prior art requires the model of the target device to be collected in advance for identification and is difficult to deploy at the edge.

[0004] In view of this, the first aspect of the present application provides a method for assisting in identifying an automatic driving drone, comprising:

[0005] Collect inspection images;

[0006] Selecting an initial image region of a target object in a first inspection image frame, and extracting target features from the initial image region of the target object in the first inspection image frame;

[0007] The filter is trained by taking the target features of the target object in the first inspection image frame as training samples, and the position of the target object in the next inspection image frame is identified by the filter.

[0008] Optionally, the filter is a filter of multiple scales, and the filter is trained using the target features of the target object in the first inspection image as training samples, including:

[0009] Determine the characteristics of the initial center position of the target object according to the target characteristics of the target object in the first inspection image frame;

[0010] Calculating the output response corresponding to the filter of each scale through the filter of each scale and the characteristics of the initial center position of the target object;

[0011] Determine the expected response of the target object in the current frame image through filters of various scales;

[0012] The training objective is to minimize the square error between the output response and the expected response corresponding to the filter of each scale, and the gradient descent method is used to train the filters of each scale.

[0013] Optionally, determining the position of the target object in the next inspection image frame by using the filter includes:

[0014] Calculate the target features of the next inspection image at each scale according to the scale of each filter;

[0015] Calculate the output response corresponding to each scale filter according to each scale filter and the target features of the next inspection image at each scale;

[0016] The corresponding position of the maximum output response in the next inspection image frame is output to obtain the position of the target object in the next inspection image frame.

[0017] Optionally, calculating the target features of the next inspection image at each scale according to the scale of each filter includes:

[0018] The target features of the next inspection image are extracted according to the scale of each filter.

[0019] Optionally, the method further includes:

[0020] The size of the tracking window is adjusted according to the size of the filter corresponding to the maximum output response.

[0021] Optionally, the tracking window size can be adjusted by:

[0022]

[0023] In the formula, To track the size of the window after adjustment, is the size of the current tracking window; is the scale of the filter corresponding to the maximum output response; is a smoothing parameter used to control the speed of scale change.

[0024] Optionally, the target features include color attribute features and gradient direction histogram features.

[0025] The second aspect of the present application provides a drone automatic driving auxiliary identification device, comprising:

[0026] A collection unit, used for collecting inspection images;

[0027] A feature extraction unit, used for selecting an initial image region of a target object in a first inspection image frame, and extracting target features from the initial image region of the target object in the first inspection image frame;

[0028] The recognition unit is used to train a filter using the target features of the target object in the first inspection image frame as training samples, and recognize the position of the target object in the next inspection image frame through the filter.

[0029] A third aspect of the present application provides an electronic device, the device comprising a processor and a memory;

[0030] The memory is used to store program code and transmit the program code to the processor;

[0031] The processor is used to execute the drone automatic driving assisted identification method described in any one of the first aspects according to the instructions in the program code.

[0032] In a fourth aspect, the present application provides a computer-readable storage medium, which is used to store program code. When the program code is executed by a processor, it implements the drone automatic driving assisted identification method described in any one of the first aspects.

[0033] It can be seen from the above technical solutions that this application has the following advantages:

[0034] The drone automatic driving assisted recognition method provided in the present application determines the initial image area of ​​the target object in the first frame of the inspection image after the inspection image is collected, and then extracts the target features of the target object from the initial image area, and trains the filter with the target features of the target object in the first frame of the inspection image as training samples, and identifies the position of the target object in the next frame of the inspection image through the filter. The method of the present application does not need to collect the model of the target device in advance, but only needs to determine the position of the target object in the first frame of the inspection image, and identifies the target object in subsequent inspection images based on the features of the target object. The method can be directly deployed at the edge. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1A flowchart of a method for assisting in identifying an autonomous driving drone provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of the structure of a drone automatic driving auxiliary identification device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] For easier understanding, see Figure 1 The present application embodiment provides a method for assisting in identifying an automatic driving drone, including:

[0040] Step 110: Collect inspection images.

[0041] Inspection images of target objects (such as power equipment, etc.) can be collected through the camera mounted on the drone.

[0042] Step 120: Select an initial image region of the target object in the first inspection image frame, and extract target features from the initial image region of the target object in the first inspection image frame.

[0043] After the UAV collects the inspection image, it can send it to the control end, and the relevant personnel will mark the initial image area of ​​the target object in the first frame of the inspection image displayed on the control end, and return the initial image area of ​​the target object to the UAV; after receiving the initial image area of ​​the target object, the UAV performs feature extraction on it, and extracts target features from the initial image area of ​​the target object in the first frame of the inspection image. The target features include color attribute features and gradient direction histogram features; the target features can be represented in the form of a circulant matrix.

[0044] Step 130: train a filter using the target features of the target object in the first inspection image frame as training samples, and identify the position of the target object in the next inspection image frame through the filter.

[0045] In one embodiment, a fixed-scale filter can be used to identify the target object. After selecting the initial image area of ​​the target object in the first frame of the inspection image, the size of the initial image area is set to the size of the tracking window; each frame of the inspection image is subjected to sliding window processing with the size of the tracking window to determine the position of the target object in each frame of the image. The filter is trained using the target features of the target object in the first frame of the inspection image as the training sample x. For the training sample x, a circulant matrix X can be constructed, and each row of this circulant matrix X is a circulant shift of the training sample x. For the circulant matrix X, its Fourier transform F(X) is a diagonal matrix, because the eigenvectors of the circulant matrix are the Fourier basis. The goal is to find a filter h to predict the ideal output y of the sample (i.e., the position of the target object). In the frequency domain, the filter h can be obtained by minimizing the energy function E:

[0046]

[0047] Where Y is the Fourier transform of the output y; is a regularization parameter used to control overfitting. By taking the derivative of the energy function E with respect to h and setting it to zero, the optimal solution for h can be obtained as:

[0048]

[0049] Where * represents the conjugate complex number; I is the identity matrix; since F(X) is a diagonal matrix, this equation can be solved element by element.

[0050] When training the filter, the present application determines the features of the initial center position of the target object according to the target features of the target object in the first inspection image. Assuming that the initial center position of the target object is (a, b), the features of the initial center position of the target object can be expressed as:

[0051]

[0052] in, is the position of the target object in the first frame image The eigenvalue at is the position of the target object in the first frame image The eigenvalue at ;

[0053] The output response y(x) is calculated by the filter and the features of the initial center position of the target object:

[0054]

[0055] Among them, K(x,y) is the filter.

[0056] The expected response of the target object in the current frame image is determined by the filter. The expected response is expressed as the expected output response of the target object in the current frame image. If the center position of the target object in the current frame image is (c, d), the expected response g(x) can be expressed as:

[0057]

[0058] The training objective is to minimize the square error between the output response and the expected response, while taking into account the regularization term to prevent overfitting. Therefore, the training objective of the filter can be expressed as:

[0059]

[0060] in, is the regularization parameter.

[0061] Using the gradient descent method to solve the filter training problem, we can get the following update formula:

[0062]

[0063] In the formula, is the learning rate;

[0064] When optimizing the training objective, the output response can be expressed as a matrix form Y=KF, and the desired response can be expressed as a matrix form G=KF, where Y is the output response matrix, G is the desired response matrix, and K is the filter matrix; therefore, the training objective of the filter can be expressed as:

[0065]

[0066] Based on the gradient descent method, the filter update formula can be expressed as:

[0067]

[0068] in, is the loss function. After the filter is trained, the filter is used to identify the position of the next inspection image according to the target features of the target object in the first inspection image.

[0069] In another embodiment, considering the use of a fixed tracking window, this may lead to performance degradation when the size of the target object changes (such as the size of the target object in the collected inspection image changes due to the distance between the drone and the target object during flight). In order to improve this problem, the present application introduces a scale estimator to dynamically adjust the size of the tracking window to estimate the scale change of the target object.

[0070] First, define a scale pool, which contains multiple predefined scale changes. , each scale s i Each corresponds to a filter The training process of multi-scale filters is as follows:

[0071] Determine the characteristics of the initial center position of the target object according to the target characteristics of the target object in the first inspection image frame;

[0072] The output response corresponding to the filter of each scale is calculated by the features of the filter of each scale and the initial center position of the target object;

[0073] Determine the expected response of the target object in the current frame image through filters of various scales;

[0074] The training objective is to minimize the square error between the output response and the expected response corresponding to the filter of each scale, and the gradient descent method is used to train the filters of each scale.

[0075] In the embodiment of the present application, for each scale s i , train an independent filter , using the same training method as the filter with a fixed tracking window, but applied to i Sample for scale The training objective of each filter is to minimize the difference between the output response and the expected response, as well as the regularization term:

[0076]

[0077] In the formula, is the expected output that is the same for all scale filters; is on scale s i The circulant matrix below; is the regularization parameter.

[0078] When identifying the target object in the next inspection image, the target features of the next inspection image at each scale are calculated according to the scale of each filter, and the target features of the next inspection image can be extracted according to the scale of each filter;

[0079] The output response corresponding to each scale filter is calculated based on the filter of each scale and the target features of the next inspection image at each scale:

[0080]

[0081] Output the corresponding position of the maximum output response in the next inspection image frame, and obtain the position of the target object in the next inspection image frame;

[0082] Find the filter scale that has the maximum output response , which is the current estimate of the scale of the most closely matching target object; adjust the size of the tracking window according to the scale of the filter corresponding to the maximum output response:

[0083]

[0084] In the formula, To track the size of the window after adjustment, is the size of the current tracking window; is the scale of the filter corresponding to the maximum output response; is a smoothing parameter used to control the speed of scale change.

[0085] The tracking window in the embodiment of the present application can adapt to the scale change of the target object, has strong robustness, can adapt to various complex environments, and improves the accuracy and robustness of tracking.

[0086] The unmanned aerial vehicle automatic driving assisted recognition method provided by the present application determines the initial image area of ​​the target object in the first frame of the inspection image after the inspection image is collected, and then extracts the target features of the target object from the initial image area, and trains the filter with the target features of the target object in the first frame of the inspection image as training samples, and identifies the position of the target object in the next frame of the inspection image through the filter. The method of the present application does not need to collect the model of the target device in advance, but only needs to determine the position of the target object in the first frame of the inspection image, and identifies the target object in the subsequent inspection images based on the features of the target object. The method can be directly deployed at the edge; and the method of the present application can dynamically adjust the scale of the tracking window, which helps to improve the tracking and recognition effect.

[0087] The above is an embodiment of a drone automatic driving assisted identification method provided by the present application, and the following is an embodiment of a drone automatic driving assisted identification device provided by the present application.

[0088] Please refer to Figure 2 , an unmanned aerial vehicle automatic driving auxiliary identification device provided in an embodiment of the present application includes:

[0089] A collection unit, used for collecting inspection images;

[0090] A feature extraction unit, used to select an initial image region of a target object in the first inspection image frame, and extract target features from the initial image region of the target object in the first inspection image frame;

[0091] The recognition unit is used to train a filter using the target features of the target object in the first inspection image frame as training samples, and recognize the position of the target object in the next inspection image frame through the filter.

[0092] The drone automatic driving assisted recognition method provided in the present application determines the initial image area of ​​the target object in the first frame of the inspection image after the inspection image is collected, and then extracts the target features of the target object from the initial image area, and trains the filter with the target features of the target object in the first frame of the inspection image as training samples, and identifies the position of the target object in the next frame of the inspection image through the filter. The method of the present application does not need to collect the model of the target device in advance, but only needs to determine the position of the target object in the first frame of the inspection image, and identifies the target object in subsequent inspection images based on the features of the target object. The method can be directly deployed at the edge.

[0093] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0094] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0095] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0096] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0097] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0098] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0099] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for executing all or part of the steps of the method described in each embodiment of the present application through a computer device (which can be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0100] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for assisting in identifying an unmanned aerial vehicle automatic driving, characterized in that: include: Collect inspection images; Selecting an initial image region of a target object in a first inspection image frame, and extracting target features from the initial image region of the target object in the first inspection image frame; The filter is trained by taking the target features of the target object in the first inspection image frame as training samples, and the position of the target object in the next inspection image frame is identified by the filter.

2. The method for assisting in identifying the automatic driving of a drone according to claim 1, characterized in that: The filter is a filter of multiple scales, and the filter is trained using the target features of the target object in the first inspection image as training samples, including: Determine the characteristics of the initial center position of the target object according to the target characteristics of the target object in the first inspection image frame; Calculating the output response corresponding to the filter of each scale through the filter of each scale and the characteristics of the initial center position of the target object; Determine the expected response of the target object in the current frame image through filters of various scales; The training objective is to minimize the square error between the output response and the expected response corresponding to the filter of each scale, and the gradient descent method is used to train the filters of each scale.

3. The method for assisting in identifying the automatic driving of a drone according to claim 2, characterized in that: Determining the position of the target object in the next inspection image frame by using the filter includes: Calculate the target features of the next inspection image at each scale according to the scale of each filter; Calculate the output response corresponding to each scale filter according to each scale filter and the target features of the next inspection image at each scale; The corresponding position of the maximum output response in the next inspection image frame is output to obtain the position of the target object in the next inspection image frame.

4. The method for assisting in identifying the automatic driving of a drone according to claim 3, characterized in that: The step of calculating target features of the next inspection image at each scale according to the scale of each filter includes: The target features of the next inspection image are extracted according to the scale of each filter.

5. The method for assisting in identifying the automatic driving of a drone according to claim 3, characterized in that: The method further comprises: The size of the tracking window is adjusted according to the size of the filter corresponding to the maximum output response.

6. The method for assisting human-machine automatic driving identification according to claim 5, characterized in that: The formula for adjusting the size of the tracking window is: In the formula, To track the size of the window after adjustment, is the size of the current tracking window; is the scale of the filter corresponding to the maximum output response; is a smoothing parameter used to control the speed of scale change.

7. The method for assisting human-machine automatic driving identification according to claim 1, characterized in that: The target features include color attribute features and gradient direction histogram features.

8. A drone automatic driving auxiliary recognition device, characterized in that: include: A collection unit, used for collecting inspection images; A feature extraction unit, used for selecting an initial image region of a target object in a first inspection image frame, and extracting target features from the initial image region of the target object in the first inspection image frame; The recognition unit is used to train a filter using the target features of the target object in the first inspection image frame as training samples, and recognize the position of the target object in the next inspection image frame through the filter.

9. An electronic device, characterized in that: The device comprises a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the drone automatic driving assisted identification method according to any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and when the program code is executed by the processor, the drone automatic driving assisted identification method according to any one of claims 1 to 7 is implemented.

Images