Multi-modal unmanned aerial vehicle visual identification method based on deep learning

Abstract

The invention belongs to the technical field of unmanned aerial vehicle visual identification, and particularly discloses an unmanned aerial vehicle visual identification method based on deep learning multi-modality, and the method comprises the steps: recognizing the visual identification information of a target object through a visual identification device carried by an unmanned aerial vehicle, the identification angle deviation degree of the unmanned aerial vehicle and the light deviation degree of each visual identification image are analyzed, so that correction angle information and the light deviation degree of the unmanned aerial vehicle are confirmed respectively, meanwhile, article information of the target article is confirmed according to the visual identification information of the target article, and corresponding feedback is carried out; the identification angle deviation degree of the unmanned aerial vehicle is analyzed according to the visual identification information of the target article, the identification accuracy of the visual identification of the unmanned aerial vehicle is guaranteed, the identification precision of the visual identification of the unmanned aerial vehicle is further guaranteed, the article information of the target article is determined according to the contours and colors of the target article and the edged articles, and the identification accuracy of the target article is improved. Therefore, the reliability of subsequent decision making is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle visual recognition, and relates to a unmanned aerial vehicle visual recognition method based on deep learning multimodality. Background Art

[0002] Drone visual recognition refers to the process of detecting, classifying and identifying target objects in the drone's environment using the visual sensor system carried by the drone through technical means such as visual recognition image processing, pattern recognition and deep learning.

[0003] For example, the Chinese invention patent with publication number CN116310910A discloses a multi-angle visual recognition method and system for unmanned aerial vehicles, including: calling a first visual recognition image acquisition device to perform visual recognition image acquisition on a first object to be identified, and obtaining a first visual recognition image acquisition result; performing dimension segmentation on the first object to be identified according to a first preset acquisition angle, obtaining a first dimension segmentation result and generating a first detection light emission angle; calling a first light sensing device to detect the first object to be identified, and obtaining a first detection result; matching the first visual recognition image acquisition result with the first dimension segmentation result, and obtaining a first matching result; traversing the first matching result based on the first spatial geometric information to perform saturation analysis, and obtaining a first analysis result including the first dimension information that does not meet the preset saturation and the first dimensional spatial geometric information, inputting the first analysis result into a visual recognition image supplement model, and obtaining a first supplementary visual recognition image; performing visual recognition image fusion processing on the first supplementary visual recognition image and the first matching result to generate a three-dimensional simulation model and add it to the first visual recognition result.

[0004] The above existing technologies have the following deficiencies: 1. When performing multi-angle visual recognition of drones, the angle deviation of drone recognition is not taken into consideration, and the recognition accuracy of drone visual recognition cannot be guaranteed, and thus the recognition accuracy of drone visual recognition cannot be guaranteed, resulting in insufficient reliability of subsequent recognition result analysis.

[0005] 2. The current analysis of visual recognition images collected by drones mainly focuses on the analysis of target objects, that is, insufficient attention is paid to other objects in the visual recognition images collected by drones, which leads to insufficient accuracy in the target object identification analysis, thereby reducing the reliability of subsequent decision-making. Summary of the invention

[0006] In view of this, in order to solve the problems raised in the above background technology, a drone visual recognition method based on deep learning multimodality is proposed.

[0007] The objective of the present invention can be achieved through the following technical solutions: The present invention provides a drone visual recognition method based on deep learning multimodality, the method comprising: step one, drone visual recognition: using a visual recognition device carried by the drone to identify each visual recognition image of the target object.

[0008] Step 2: Analysis of drone recognition angles: Analyze the deviation of the drone's recognition angles based on the visual recognition images of the target object.

[0009] Step 4: Confirm the drone angle correction: Confirm the drone's correction angle information based on the drone's identification angle deviation.

[0010] Step 5: Drone recognition light analysis: Based on the visual recognition images of the target object, analyze the light deviation of each visual recognition image.

[0011] Step 5: Confirmation of drone recognition light: Confirm the drone's light deviation based on the light deviation of each visual recognition image.

[0012] Step 6: Confirmation of drone recognition target: Based on the visual recognition images of the target object, the historical information of the drone is extracted from the drone recognition cloud platform to confirm the object information of the target object.

[0013] Step 7: Drone identification feedback: Provide corresponding feedback based on the drone's correction angle information, light deviation, and target object information.

[0014] Compared with the prior art, the present invention has the following beneficial effects: (1) By analyzing the recognition angle deviation of the drone according to each visual recognition image of the target object, the present invention avoids the current deficiency of not considering the recognition angle deviation of the drone, ensures the recognition accuracy of the drone visual recognition, and further ensures the recognition precision of the drone visual recognition, thereby improving the reliability of subsequent recognition result analysis. At the same time, on the other hand, by considering the recognition angle deviation of the drone, the application of drone visual recognition in complex scenes is realized.

[0015] (2) The present invention confirms the object information of the target object based on the outline and color of the target object and each border object, thereby breaking the current problem of insufficient attention to other objects in the visual recognition image collected by the drone, thereby ensuring the accuracy of the target object recognition analysis, thereby improving the reliability of subsequent decision-making.

[0016] (3) The present invention analyzes the light deviation of each visual recognition image according to the brightness qualification index, contrast qualification index and color saturation qualification index, and comprehensively evaluates the light deviation from multiple key dimensions, thereby avoiding the limitation of judging the light deviation from only a single factor, and improving the validity of the light deviation analysis results. At the same time, by setting the weights of the brightness qualification index, contrast qualification index and color saturation qualification index, the stability and accuracy of the light deviation analysis results are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0018] Figure 1 It is a schematic diagram of the connection of each step of the method of the present invention.

[0019] Figure 2 It is a connection diagram of the correction angle information confirmation step of the present invention.

[0020] Figure 3 It is a connection diagram of the item information confirmation step of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] See also Figure 1 As shown, the present invention provides a drone visual recognition method based on deep learning multimodality, the method comprising: step one, drone visual recognition: identifying each visual recognition image of the target object through a visual recognition device carried by the drone.

[0023] It should be added that each visual recognition image of the target object is obtained by taking pictures with a camera mounted on the drone.

[0024] It should be supplemented that the information extracted from each visual recognition image of the target object includes: the angle between the target object reference line and the horizontal reference line in each visual recognition image, the angle between the target object reference line and the vertical reference line in each visual recognition image, the average brightness of each visual recognition image, and the outline and color of the target object and each border object.

[0025] It should be added that the angle between the target object reference line and the horizontal reference line in each visual recognition image is obtained in the following way: the horizontal reference line is determined by the coordinates of the visual recognition image. In the visual recognition image of the Cartesian coordinate system, the axis direction can usually be used as the horizontal reference line. If the visual recognition image is taken at an angle, the horizontal reference line is determined by finding the horizon in the visual recognition image or the edge of a known horizontal object.

[0026] If the target object reference line and the horizontal reference line are both straight lines, and the slope of the target object reference line is , the slope of the horizontal reference line is (Since the slope of a horizontal line is 0), the angle between the two lines can be calculated by the formula Calculate the angle using the inverse tangent function.

[0027] If the target object reference line is a curve, you need to first select several points on the curve, calculate the slope of the tangent at these points, and then calculate the angle between each tangent and the horizontal reference line according to the straight line-straight line method. Finally, get the angle value by taking the average value or weighted average according to the importance of each point on the curve.

[0028] It should be added that the method for obtaining the angle between the target object reference line and the vertical reference line in each visual recognition image is similar to the method for obtaining the angle between the target object reference line and the horizontal reference line in each visual recognition image mentioned above, and will not be repeated.

[0029] It should be added that the average brightness of each visual recognition image is obtained by giving red ( ),green( ),blue( ) different weights to calculate brightness, the weights usually used are (This is based on the principle that the human eye is most sensitive to green, followed by red, and least sensitive to blue). The steps to calculate the average brightness are: Calculate the brightness of each pixel according to the above formula , It is the pixel number of the visual recognition image height. , The pixel number for the visual recognition image width, , and then calculate the average brightness of each visual recognition image: , is the number of pixels of the visual recognition image height, The width of the visual recognition image in pixels.

[0030] It should be added that the target object contour is obtained by using the Canny edge detection algorithm to monitor the edge and obtain a set of edge pixels. The target object contour is then constructed based on the position and connection relationship of the edge pixels.

[0031] It should be added that the color of the target object is obtained as follows: if the color of the target object is relatively single or uniform, the average value of the pixels in the target object area is calculated as the color representative of the area. If the color distribution of the target object is more complex, a color histogram of the target object area is constructed. The histogram can reflect the frequency of occurrence of different colors in the visual recognition image, and then the main color of the target object is determined by analyzing the peak value and distribution of the histogram. For example, if a certain value range of a color channel in the histogram shows an obvious peak, then this color is taken as the main color of the target object.

[0032] It should be added that the border objects of the target object refer to the objects next to the target object in the visual recognition image of the target object. The method for obtaining each border object in the target object is: to obtain them in the same way as the method for obtaining the outline and color of the target object.

[0033] Step 2: Analysis of drone recognition angles: Analyze the deviation of the drone's recognition angles based on the visual recognition images of the target object.

[0034] Exemplarily, the analyzing the recognition angle deviation of the drone includes: extracting the angle between the target object reference line and the horizontal reference line from each visual recognition image of the target object.

[0035] The angle between the target object reference line and the horizontal reference line in each visual recognition image is subtracted from the standard angle between the object reference line and the horizontal reference line to obtain the horizontal deviation angle of each visual recognition image, which is recorded as , Number the visual recognition images, .

[0036] From each visual recognition image of the target object, extract the angle between the reference line of the target object in each visual recognition image and the vertical reference line, and then The vertical deviation angle of each visual recognition image is obtained by the same analysis method, which is recorded as .

[0037] Count the comprehensive deviation angles of each visual recognition image , .

[0038] Statistical average deviation angle of visual recognition images , , is the number of visual recognition images.

[0039] Statistical visual recognition image standard deviation angle , .

[0040] In a specific embodiment, if there are 10 visual recognition images, and their comprehensive deviation angles are 4°, 5°, 3°, 6°, 4°, 5°, 4°, 3°, 5°, and 4° respectively, then the average deviation angle of the visual recognition images is .

[0041] The standard deviation angle of the visual recognition image is .

[0042] Statistical analysis of the drone's recognition angle deviation , , The standard deviation angle for setting the reference.

[0043] It should be added that It is extracted from the UAV visual recognition technical specification table.

[0044] The embodiment of the present invention analyzes the recognition angle deviation of the drone according to each visual recognition image of the target object, thereby avoiding the current deficiency of not considering the recognition angle deviation of the drone, ensuring the recognition accuracy of the drone visual recognition, and further ensuring the recognition precision of the drone visual recognition, thereby improving the reliability of subsequent recognition result analysis. At the same time, on the other hand, by considering the recognition angle deviation of the drone, the application of drone visual recognition in complex scenes is realized.

[0045] Step 4: Confirm the drone angle correction: Confirm the drone's correction angle information based on the drone's identification angle deviation.

[0046] See also Figure 2 As shown, exemplarily, the confirmation of the correction angle information of the drone includes: E1. If the recognition angle deviation of the drone is less than or equal to the set recognition angle deviation, no correction is required as the correction angle information of the drone.

[0047] E2. If the recognition angle deviation of the drone is greater than the set recognition angle deviation, the horizontal deviation angle of each visual recognition image and the vertical deviation angle of each visual recognition image are averaged to obtain the average horizontal deviation angle and the average vertical deviation angle of the visual recognition image.

[0048] E3. If the horizontal deviation angle of the visual recognition image is less than 0, the horizontal correction direction of the UAV is to the right, otherwise, the horizontal correction direction of the UAV is to the left, thereby obtaining the horizontal correction direction of the UAV.

[0049] E4. If the average vertical deviation angle of the visual recognition image is less than 0, the vertical correction direction of the UAV is upward correction. Otherwise, the vertical correction direction of the UAV is downward correction, and the vertical correction direction of the UAV is obtained.

[0050] E5. The absolute values ​​of the average horizontal deviation angle and the average vertical deviation angle of the visual recognition image are used as the horizontal correction angle and the vertical correction angle of the UAV respectively.

[0051] E6. The horizontal correction direction, vertical correction direction, horizontal correction angle and vertical correction angle of the drone are used as the correction angle information of the drone.

[0052] Step 5: Drone recognition light analysis: Based on the visual recognition images of the target object, analyze the light deviation of each visual recognition image.

[0053] Exemplarily, the analysis of the light deviation of each visual recognition image includes: F1, extracting the average brightness of each visual recognition image from each visual recognition image of the target object, and then calculating the brightness deviation of each visual recognition image. .

[0054] Furthermore, the counting of brightness deviations of the visual recognition images includes: F1-1, if the average brightness of a visual recognition image is within a set standard brightness range of the visual recognition image, the brightness deviation of the visual recognition image is recorded as 0.

[0055] F1-2. If the average brightness of a visual recognition image is less than the lower limit of the standard brightness range of the visual recognition image, the average brightness of the visual recognition image and the lower limit of the standard brightness range of the visual recognition image are recorded as and .

[0056] F1-3. Count the brightness deviation of the visual recognition image , .

[0057] F1-4. If the average brightness of a visual recognition image is greater than the upper limit of the standard brightness range of the visual recognition image, the upper limit of the standard brightness range of the visual recognition image is recorded as .

[0058] F1-5. Count the brightness deviation of the visual recognition image , , and then the brightness deviation of each visual recognition image is obtained and recorded as , The value is 0 or or .

[0059] In a specific embodiment, the standard brightness range of the visual recognition image is [100, 140]. If the average brightness of a visual recognition image is 115, the brightness deviation of the visual recognition image is 0. If the average brightness of a visual recognition image is 80, which is lower than the lower limit of the standard brightness range of the visual recognition image, the brightness deviation of the visual recognition image is , if the average brightness of a visual recognition image is 150, which is higher than the upper limit of the standard brightness range of the visual recognition image, then the brightness deviation of the visual recognition image is .

[0060] F2. Extract the contrast and color saturation of each visual recognition image from each visual recognition image of the target object, and then obtain the contrast deviation and color saturation deviation of each visual recognition image in the same way as the statistical method of the brightness deviation of each visual recognition image, and record them as and .

[0061] F3. Count the light deviation of each visual recognition image , , , and are the weights of the set brightness deviation, contrast deviation and color saturation deviation, respectively. , .

[0062] It should be added that brightness determines the visibility of objects in visual recognition images. If the brightness is too low, the visual recognition image will be too dim, and the target object may be difficult to be clearly identified or even completely invisible. For example, in visual recognition images taken at night or in low-light environments, if the brightness is insufficient, the shape and features of the target object may not be distinguished. If the brightness is too high, the visual recognition image may be overexposed, resulting in loss of details of the target object. Contrast determines the degree of difference between different areas in the visual recognition image. Appropriate contrast can better separate the target object from the background and improve the recognition of the target object. For example, a black object against a white background will be very prominent if the contrast is high, but it may blend into the background and be difficult to distinguish if the contrast is low. However, its impact on the visibility of the object is less than that of the brightness, so the weight of the brightness deviation is greater than that of the contrast deviation. Color saturation is not a decisive factor in the recognition of the target object. Even if the color saturation is low, the target object can still be recognized as long as the brightness and contrast are appropriate. Color saturation affects the aesthetics and visual effect of the visual recognition image more, but has relatively little impact on the actual recognition accuracy of the target object. Therefore, setting , in order to facilitate analysis, The specific value can be 0.5. The specific value can be 0.3. The specific value can be 0.2.

[0063] The embodiment of the present invention analyzes the light deviation of each visual recognition image according to the brightness qualification index, contrast qualification index and color saturation qualification index, and comprehensively evaluates the light deviation from multiple key dimensions, thereby avoiding the limitation of judging light deviation from only a single factor, and improving the validity of the light deviation analysis results. At the same time, by setting the weights of the brightness qualification index, contrast qualification index and color saturation qualification index, the stability and accuracy of the light deviation analysis results are improved.

[0064] Step 5: Confirmation of drone recognition light: Confirm the drone's light deviation based on the light deviation of each visual recognition image.

[0065] Exemplarily, determining the light deviation of the drone includes: Perform mean calculation to obtain the average light deviation of the visual recognition image, recorded as .

[0066] In a specific embodiment, there are 5 visual recognition images, whose light deviations are 4.1, 3.8, 4.3, 3.9, and 4.2 respectively. Then the average light deviation of the visual recognition images is .

[0067] Will and Importing the standard deviation formula, we get the standard light deviation of the visual recognition image, recorded as .

[0068] In a specific embodiment, the calculation process of the standard light deviation of the visual recognition image is as follows: the light deviations are 4.1, 3.8, 4.3, 3.9, and 4.2 respectively, and the average light deviation of the visual recognition image is , then calculate the square of the difference between the light deviation of each visual recognition image and the average light deviation of the visual recognition image: , , , , , and then calculate the mean of the squares: , thereby calculating the standard light deviation of the visual recognition image .

[0069] Counting the light deviation of the drone , , and They are the average light deviation and standard light deviation of the reference respectively.

[0070] It should be added that and All are extracted from the UAV visual recognition technical specification table.

[0071] Step 6: Confirmation of drone recognition target: Based on the visual recognition images of the target object, the historical information of the drone is extracted from the drone recognition cloud platform to confirm the object information of the target object.

[0072] It should be added that the parameter information of each historically identified object includes: category information, outline, border objects, and hue, saturation, and brightness of each historically identified object.

[0073] See also Figure 3 As shown, exemplarily, the confirming the object information of the target object includes: H1, extracting parameter information of each historically identified object from the historical information of the drone.

[0074] H2: Extract the outline of the target object from each visual recognition image of the target object, and then confirm the similarity between the outline of the target object and each historical recognition object. , Item number for historical identification, .

[0075] Furthermore, the determination of the similarity between the contours of the target object and each historically identified object includes: H2-1, extracting the perimeter of the target object from the contour of the target object, recorded as , and extract the contour of each historically identified object from the parameter information of each historically identified object, and then extract the perimeter of each historically identified object, recorded as .

[0076] H2-2. Count the similarities between the perimeter of the target object and each historically identified object , .

[0077] H2-3, extract the volume of the target object from the contour of the target object, and extract the volume of each historically identified object from the contour of each historically identified object, and then The volume similarity between the target object and each historically identified object is obtained by the statistical method of .

[0078] H2-4. Extract each contour feature point of the target object from the contour of the target object, and extract each contour feature point of each historically identified object from the contour of each historically identified object, and then match and compare each contour feature point of the target object with each contour feature point of each historically identified object to obtain the number of matching contour points of the target object and each historically identified object, and then use the ratio of the number of matching contour feature points of the target object to the number of contour feature points of the target object as the shape factor of the target object and each historically identified object, recorded as .

[0079] H2-5. Count the similarities between the target object and the contours of each historically identified object , .

[0080] In a specific embodiment, the perimeter of the target object is 30 cm and the volume is 50 cubic centimeters, and the perimeter of the historical object A is 28 cm and the volume is 45 cubic centimeters. Then the perimeter similarity between the target object and the historical identification object A is , the volume similarity between the target object and the historically recognized object A is .

[0081] To further calculate the shape factor, the SIFT algorithm is used to extract the contour feature points of the target object and historical object A. The number of feature points of the target object and historical object A is 100 and 90, respectively. After matching, 80 matching feature points are found, so the shape factor is .

[0082] Count the similarity between the target object and the historically identified object A , .

[0083] H3: Extract the color of the target object from each visual recognition image of the target object, and then confirm the color similarity between the target object and each historical recognition object .

[0084] Furthermore, the color similarity between the target object and each historically identified object is confirmed, including: H3-1, extracting the hue, saturation and brightness of the target object color from the color of the target object, recorded as , and .

[0085] H3-2. Extract the hue, saturation and brightness of the color of each historically identified object from the parameter information of each historically identified object, and record them as , and , and then use the Euclidean distance to calculate the color similarity distance between the target object and each historically identified object , .

[0086] H3-3. Count the color similarities between the target object and each historically identified object , , Sets the color similarity distance for reference.

[0087] It should be added that It is extracted from the UAV visual recognition technical specification table.

[0088] H4. Import the contour similarity and color similarity between the target object and each historically identified object into the similarity analysis model to obtain the similarity between the target object and each historically identified object.

[0089] It should be added that the specific analysis formula of the similarity analysis model is: Statistical similarity between the target item and each historically identified item , , and are the contour similarity and color similarity weights set respectively, , .

[0090] It should be added that the outline is the basic shape feature of an object, which can provide key information about the overall structure and form of the object. In the process of visual cognition, the outline is often one of the first information processed by the brain to quickly distinguish the category of the object. For target objects and historically identified objects, high outline similarity means that they have a strong correlation in basic shape and are likely to belong to the same type of object. Color is an important auxiliary feature in visual recognition. Although it is not as critical as the definition of the object as the outline, color can help further refine the recognition of the object. Objects with the same outline may belong to different subspecies or have different uses due to different colors. For example, a red fire truck and a yellow school bus may have similar outlines, but the difference in color gives them different functions and identification features, so setting , in order to facilitate analysis, The specific value can be 0.6. The specific value can be 0.4.

[0091] H5. Filter out the historically identified item with the greatest similarity from the similarities between the target item and each historically identified item, and extract the category of the historically identified item from the parameter information of each historically identified item as the historically identified category.

[0092] H6. Extract each border item of the target object from each visual recognition image of the target object, confirm the border recognition category information, and extract the border recognition category therefrom.

[0093] Furthermore, the confirming of the border recognition category information includes: H6-1, extracting the outline and color of each border object from each visual recognition image of the target object.

[0094] H6-2. The similarity between the target object and each historically identified object is statistically obtained in the same way as the statistical method of the similarity between the target object and each historically identified object, and the border object with the maximum value is selected, and then the category information of the border object is extracted from the drone recognition cloud platform as the border recognition category information.

[0095] H7. If the historical identification category is consistent with the border identification category, the category information of the historical identification category is extracted from the parameter information of each historically identified item as the item information of the target item; otherwise, the border identification category information is used as the item information of the target item.

[0096] The embodiment of the present invention confirms the object information of the target object according to the outline and color of the target object and each border object, thereby breaking the current problem of insufficient attention to other objects in the visual recognition image collected by the drone, thereby ensuring the accuracy of the target object recognition analysis, thereby improving the reliability of subsequent decision-making.

[0097] Step 7: Drone identification feedback: Provide corresponding feedback based on the drone's correction angle information, light deviation, and target object information.

[0098] The above contents are merely examples and explanations of the concept of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, they shall all fall within the protection scope of the present invention.

Claims

1. A multi-modal drone visual recognition method based on deep learning, characterized by: The method includes: Step 1: UAV visual recognition: Use the visual recognition equipment on the drone to identify the visual recognition images of the target object; Step 2: Analysis of drone recognition angles: Analyze the deviation of the drone's recognition angles based on the visual recognition images of the target object; Step 3: Confirm the angle correction of the drone: confirm the correction angle information of the drone according to the angle deviation of the drone recognition; Step 4: UAV recognition light analysis: Analyze the light deviation of each visual recognition image based on each visual recognition image of the target object; Step 5: Confirmation of drone recognition light: Confirm the drone’s light deviation based on the light deviation of each visual recognition image; Step 6: Confirmation of drone recognition target: Based on the visual recognition images of the target object, the historical information of the drone is extracted from the drone recognition cloud platform to confirm the object information of the target object; Step 7: Drone identification feedback: Provide corresponding feedback based on the drone's correction angle information, light deviation, and target object information.

2. The method for drone visual recognition based on deep learning multimodality according to claim 1, characterized in that: The analysis of the recognition angle deviation of the drone includes: extracting an angle between a target object reference line and a horizontal reference line from each visual recognition image of the target object; The angle between the target object reference line and the horizontal reference line in each visual recognition image is subtracted from the standard angle between the object reference line and the horizontal reference line to obtain the horizontal deviation angle of each visual recognition image, which is recorded as , Number the visual recognition images, ; From each visual recognition image of the target object, extract the angle between the reference line of the target object in each visual recognition image and the vertical reference line, and then The vertical deviation angle of each visual recognition image is obtained by the same analysis method, which is recorded as ; Count the comprehensive deviation angles of each visual recognition image , ; Statistical average deviation angle of visual recognition images , , is the number of images for visual recognition; Statistical visual recognition image standard deviation angle , ; Statistical analysis of the drone's recognition angle deviation , , The standard deviation angle for setting the reference.

3. The method for drone visual recognition based on deep learning multimodality according to claim 2, characterized in that: The confirmation of the correction angle information of the drone includes: E1. If the recognition angle deviation of the drone is less than or equal to the set recognition angle deviation, no correction is required as the correction angle information of the drone; E2. If the recognition angle deviation of the drone is greater than the set recognition angle deviation, the horizontal deviation angle of each visual recognition image and the vertical deviation angle of each visual recognition image are averaged to obtain the average horizontal deviation angle and the average vertical deviation angle of the visual recognition image; E3. If the average horizontal deviation angle of the visual recognition image is less than 0, the horizontal correction direction of the drone is to the right, otherwise, the horizontal correction direction of the drone is to the left, and then the horizontal correction direction of the drone is obtained; E4. If the average vertical deviation angle of the visual recognition image is less than 0, the vertical correction direction of the drone is upward correction. Otherwise, the vertical correction direction of the drone is downward correction, and the vertical correction direction of the drone is obtained; E5. The absolute values ​​of the average horizontal deviation angle and the average vertical deviation angle of the visual recognition image are used as the horizontal correction angle and the vertical correction angle of the drone respectively; E6. The horizontal correction direction, vertical correction direction, horizontal correction angle and vertical correction angle of the drone are used as the correction angle information of the drone.

4. The method for drone visual recognition based on deep learning multimodality according to claim 1, characterized in that: The analyzing the light deviation of each visual recognition image includes: Extract the average brightness of each visual recognition image from each visual recognition image of the target object, and then calculate the brightness deviation of each visual recognition image ; The contrast and color saturation of each visual recognition image are extracted from each visual recognition image of the target object, and then the contrast deviation and color saturation deviation of each visual recognition image are obtained by statistically analyzing the brightness deviation of each visual recognition image in the same way, and are recorded as and ; Count the light deviation of each visual recognition image , , , and are the weights of the set brightness deviation, contrast deviation and color saturation deviation, respectively. , .

5. The method for drone visual recognition based on deep learning multimodality according to claim 4, characterized in that: The counting of brightness deviations of the visual recognition images includes: If the average brightness of a visual recognition image is within the set standard brightness range of the visual recognition image, the brightness deviation of the visual recognition image is recorded as 0; If the average brightness of a visual recognition image is less than the lower limit of the standard brightness range of the visual recognition image, the average brightness of the visual recognition image and the lower limit of the standard brightness range of the visual recognition image are recorded as and ; Count the brightness deviation of the visual recognition image , ; If the average brightness of a visual recognition image is greater than the upper limit of the standard brightness range of the visual recognition image, the upper limit of the standard brightness range of the visual recognition image is recorded as ; Count the brightness deviation of the visual recognition image , , and then the brightness deviation of each visual recognition image is obtained and recorded as , The value is 0 or or .

6. The method for drone visual recognition based on deep learning multimodality according to claim 4, characterized in that: The step of confirming the light deviation of the drone includes: Will Perform mean calculation to obtain the average light deviation of the visual recognition image, recorded as ; Will and Importing the standard deviation formula, we get the standard light deviation of the visual recognition image, recorded as ; Counting the light deviation of the drone , , and They are the average light deviation and standard light deviation of the reference respectively.

7. The method for drone visual recognition based on deep learning multimodality according to claim 1, characterized in that: The confirmed target item information includes: H1. Extract parameter information of each historically identified object from the historical information of the drone; H2: Extract the outline of the target object from each visual recognition image of the target object, and then confirm the similarity between the outline of the target object and each historical recognition object. , Item number for historical identification, ; H3: Extract the color of the target object from each visual recognition image of the target object, and then confirm the color similarity between the target object and each historical recognition object ; H4, importing the contour similarity and color similarity between the target object and each historically identified object into the similarity analysis model to obtain the similarity between the target object and each historically identified object; H5. Filter out the historically identified item with the greatest similarity from the similarities between the target item and each historically identified item, and extract the category of the historically identified item from the parameter information of each historically identified item as the historically identified category; H6, extracting each border object of the target object from each visual recognition image of the target object, further confirming the border recognition category information, and extracting the border recognition category therefrom; H7. If the historical identification category is consistent with the border identification category, the category information of the historical identification category is extracted from the parameter information of each historically identified item as the item information of the target item; otherwise, the border identification category information is used as the item information of the target item.

8. The method for drone visual recognition based on deep learning multimodality according to claim 7, characterized in that: The step of confirming the similarity of the outlines of the target object and each historically identified object includes: Extract the perimeter of the target object from its contour, denoted as , and extract the contour of each historically identified object from the parameter information of each historically identified object, and then extract the perimeter of each historically identified object, recorded as ; Count the similarity of the perimeter of the target object and each historically identified object , ; The volume of the target object is extracted from the contour of the target object, and the volume of each historically identified object is extracted from the contour of each historically identified object, and then The volume similarity between the target object and each historically identified object is obtained by the statistical method of ; The contour feature points of the target object are extracted from the contour of the target object, and the contour feature points of each historically identified object are extracted from the contour of each historically identified object. Then, the contour feature points of the target object are matched and compared with the contour feature points of each historically identified object to obtain the number of matching contour points of the target object and each historically identified object. The ratio of the number of matching contour feature points to the number of contour feature points of the target object is used as the shape factor of the target object and each historically identified object, which is recorded as ; Count the similarities between the target object and the contours of each historically identified object , .

9. The method for drone visual recognition based on deep learning multimodality according to claim 7, characterized in that: The confirming of the color similarity between the target object and each historically identified object includes: Extract the hue, saturation and brightness of the target object’s color from the color of the target object, denoted as , and ; The hue, saturation and brightness of the color of each historically identified object are extracted from the parameter information of each historically identified object and recorded as , and , and then use the Euclidean distance to calculate the color similarity distance between the target object and each historically identified object , ; Count the color similarity between the target object and each historically identified object , , Sets the color similarity distance for reference.

10. The method for drone visual recognition based on deep learning multimodality according to claim 7, characterized in that: The confirmation border identification category information includes: Extracting the outline and color of each border object from each visual recognition image of the target object; The similarity between the target object and each historically identified object is statistically obtained in the same way as the statistical method of the similarity between the target object and each border object, and the border object with the maximum value is screened out, and then the category information of the border object is extracted from the drone recognition cloud platform as the border recognition category information.

Images