Unmanned aerial vehicle image processing and target identification system and method based on deep learning

Abstract

The invention relates to the technical field of image processing, in particular to an unmanned aerial vehicle image processing and target recognition system and method based on deep learning, and the method specifically comprises the steps: collecting greenhouse structure parameters, plant distribution density and unmanned aerial vehicle flight data, building a plant-airflow coupling equation, and outputting an image effective window period; the humidity of the greenhouse is monitored in real time, a humidity-reflection correlation compensation model is established for deformation compensation, and a distortion correction image after reflection suppression is output; based on the distortion correction image, through GAN restoration under topological constraint, processing a plant-shielded area in the image, and outputting a target object candidate image after the shielded area is restored; the target object state is evaluated, and quantitative indexes are provided for work task decision making. According to the invention, the problem of low dynamic target object recognition accuracy in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of online monitoring, and is an unmanned aerial vehicle (UAV) image processing and target recognition system and method based on deep learning. Background Art

[0002] In greenhouse pollination operations, UAV image processing technology faces the following technical problems, including: Most existing airflow disturbance compensation methods are based on the assumption of a uniform flow field and fail to consider the coupling effect of the jet flow at the ventilation opening and the plant distribution in the closed greenhouse environment, resulting in a large prediction error in the image acquisition window period and seriously affecting the capture accuracy of dynamic targets. At the same time, existing anti-reflection strategies cannot adapt to the wavelength-selective reflection caused by the formation of a water film on the surface of petals in the high-humidity greenhouse environment. In addition, existing occlusion repair methods rely on general generative adversarial networks and lack modeling of the topological growth law of plants, resulting in structural errors that violate botanical morphology in the repaired areas. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to propose an unmanned aerial vehicle image processing and target recognition system and method based on deep learning for the problem of low recognition accuracy of dynamic target objects in the prior art.

[0004] In order to achieve the above object, the technical solution of the unmanned aerial vehicle image processing and target recognition method based on deep learning of the present invention includes the following steps: Step 1: Collect greenhouse structure parameters, plant distribution density, and UAV flight data, establish a plant-airflow coupling equation, and output an effective image window period; Step 2: Real-time monitor the humidity in the greenhouse, establish a humidity-reflection correlation compensation model for deformation compensation, and output a distortion-corrected image after anti-reflection; Step 3: Based on the distortion-corrected image, process the area occluded by plants in the image through GAN repair under topological constraints, and output a candidate map of the target object after repairing the occluded area; Step 4: Evaluate the state of the target object and provide a quantitative index for work task decision-making.

[0005] Specifically, Step 1 includes: A11: Collect the three-dimensional model of the greenhouse structure and the plant planting layout map, and calibrate the greenhouse microenvironment parameters, where the greenhouse microenvironment parameters include: average wind speed and plant distribution turbulence intensity value; Among them, the three-dimensional model of the greenhouse structure includes: ventilation opening coordinate data; the plant planting layout map includes: row spacing data and plant spacing data; A12: Obtain the characteristic data of the plant where the target object is located and the rotor speed of the UAV, calibrate the bending stiffness of the plant stem through a three-point bending dynamic experiment, and at the same time, calibrate the mechanical parameters of the plant by establishing a non-uniform force distribution model. The calibration strategy for the bending stiffness of the stalk is as follows: : ; where d is the diameter of the stalk; is the concentrated force applied in the three-point bending dynamic experiment; is the support span; is the maximum mid-span deflection; The non-uniform force distribution model is specifically: ; is the distributed aerodynamic load applied by the downwash airflow of the UAV rotor to the plant stalk; is the force conversion coefficient; is the rotor speed of the UAV; X is the longitudinal position along the plant stalk; is the length of the plant stalk.

[0006] Specifically, step one further includes: A21: Based on A11 - A12, establish a real-time prediction equation for plant swing, and predict and obtain the displacement of the target object on the plant ; The real-time prediction equation for plant swing is specifically: ; where are the stalk density and cross-sectional area respectively; c is the damping coefficient; A22: According to the displacement of the target object on the plant and the sampling frequency of the UAV camera, calculate the real-time swing phase of the target object, determine the effective image window period, and output the effective sampling window ; The determination of the effective image window period includes: ; where is the real-time swing phase of the target object.

[0007] Preferably, the calculation formula for the real-time swing phase of the target object is: ; where is the velocity changing with time t at the position of the flower stamen height ; represents the displacement at the position of the flower stamen height ; Specifically, step two includes: B11: Fit the relationship between the humidity reflectance correction coefficients corresponding to each band and the greenhouse humidity. Using the least squares method, adjust the parameters in the relationship formula to minimize the error between the theoretical reflectance and the measured reflectance, and output the humidity-reflectance correction coefficient with the minimum error. , and correct the reflectance according to the humidity-reflectance correction coefficient to obtain the corrected reflectance. ; Preferably, the correction of the reflectance includes: ; where are the corrected reflectance and the reflectance before correction, respectively; B12: Acquire the original polarization image through a polarization camera at different polarization angles. By traversing different polarization angles, find the angle that minimizes the adjusted light intensity as the optimal polarization angle. ; Preferably, the acquisition strategy of the optimal polarization angle is: ; where is the light intensity of the original polarization image at the polarization angle of ; is the reflectance in the current environment; B13: Update the optimal polarization angle every 10 frames through the dynamic adjustment strategy of the polarization angle. The specific dynamic adjustment strategy of the polarization angle is: ; where are the polarization angles at the next moment and the previous moment; is the flower region mask; (i, j) represents a pixel point, represents the intensity value of the image at the pixel point (i, j); represents the total reflection intensity of the region; It should be noted that the flower region mask is obtained through an image segmentation algorithm to ensure that only the reflected light of the flower region is optimized.

[0008] It should be noted that the dynamic adjustment strategy of the polarization angle is based on the gradient descent algorithm. By calculating the gradient of the reflected light intensity of the flower region with respect to the polarization angle, the polarization angle is gradually adjusted with a step size of 0.1 to gradually reduce the reflected light intensity of the flower region and reach the optimal state; It should be noted that in the actual greenhouse environment, humidity will change the reflectance of the petals, thereby affecting the reflected light intensity; It should be noted that by optimizing the polarization light field, the interference of specular reflected light on the petal surface is reduced, and the image quality is improved.

[0009] B14: Extract the effective window period of the image and the angular velocity information of the drone , the optimal shutter trigger moment for image capture , and capture the original image at the optimal shutter trigger moment ; Preferably, the calculation strategy for the optimal shutter trigger moment is as follows: ; wherein, is the current moment, is the current swing phase.

[0010] Specifically, step two further includes: extracting the original image and the stem bending stiffness matrix, calculating the actual displacement of each control point according to the stiffness matrix and the swing displacement, and applying a reverse displacement to each pixel through TPS transformation to restore the stem to a vertical state and obtain a distortion-corrected image; Preferably, the calculation formula for the displacement amount of each control point is: wherein, is the local stiffness, and Sw is the displacement caused by the plant swing; Calculate the distance from this pixel point to each control point Substitute it into the kernel function; wherein, the kernel function is , which is used to describe the influence degree of the control point on the surrounding pixels; Preferably, the acquisition strategy for the corrected image is: ; is the corrected image; (x, y) is the image pixel coordinate, is the coordinate of the control point set along the stem, with a total of N, and n is the label index of the control point. One control point is set every 5 cm along the stem; is the weight coefficient, which is solved through stiffness constraints, and the weight will be automatically allocated according to the control point displacement and stiffness, and the weight of the control point closer to the pixel is greater; Specifically, step three includes: C11: Obtain the row spacing data and layer height data of the plant planting, and at the same time collect the three-dimensional coordinate information of each point of the plant. Remove the noise points through preprocessing, retain the effective depth point cloud data, and construct a plant topological skeleton diagram according to the effective depth point cloud data; C12: Obtain the plant topological skeleton diagram and use a generative adversarial network for occlusion repair. The occlusion repair includes: Concatenate the noise vector and the plant topological skeleton diagram in the channel dimension as the input of the generator; Through the topological attention module, use the convolutional layer to generate a query vector Q from the skeleton diagram to capture structural features, and the convolutional layer generates a key vector K and a value vector V from the occluded image to focus on the visual features of the occluded area, enabling the generator to repair the occluded area under the guidance of the skeleton; Input the real image and the repaired image into the discriminator. Use the pre-trained U-Net model SkelNet to extract the skeleton, and calculate the topological consistency loss through the L1 norm , combined with the feature extraction layer of the VGG network, calculate the perceptual loss through the L2 norm ; Calculate the total loss LOSS, and the calculation strategy of the total loss is: ; Among them, is the adversarial loss; It should be noted that the adversarial loss prompts the generator to generate realistic images; C13: Obtain the image after occlusion repair.

[0011] Specifically, step three further includes: C21: Convert the original image and the repaired image to the LAB color space and calculate the color difference value. The calculation strategy of the color difference value is: ; Among them, is the original image; is the image after occlusion repair; is the color space conversion function; is the L2 norm; It should be noted that the color space conversion function is used to convert the input image from color spaces such as RGB to the CIELAB color space; It should be noted that the L2 norm is used to calculate the square root of the sum of the squares of the corresponding elements of two vectors C22: Preset the color difference standard threshold, traverse all the repaired areas in the repaired image. When the color difference value of the current area is less than the color difference standard threshold, judge that the current area is a reliable repaired area with consistent illumination and retain the repair; When the color difference value of the current area is greater than or equal to the color difference standard threshold, judge that the current area is an unreliable repaired area with inconsistent illumination and revoke the repair; C23: Output the candidate map of the target object after occlusion area repair.

[0012] In addition, the UAV image processing and target recognition system based on deep learning of the present invention includes the following modules: A window period determination module, a distortion correction module, an occlusion repair module, a task decision module, and a main control module; The window period determination module is used to collect greenhouse structure parameters, plant distribution density, and UAV flight data, establish a plant-airflow coupling equation, and output an effective image window period; The distortion correction module is used to monitor the greenhouse humidity in real time, establish a humidity-reflection correlation compensation model for deformation compensation, and output a distortion-corrected image after reflection suppression; The occlusion repair module is used to process the plant occlusion area in the image through GAN repair under topological constraints based on the distortion-corrected image, and output a target candidate map after occlusion area repair; The task decision module is used to evaluate the state of the target and provide a quantitative index for work task decision-making; The main control module is used to run and control other modules.

[0013] Compared with the prior art, the technical effects of the present invention are as follows: By establishing a plant-airflow coupling model and considering the dynamic disturbance compensation of the interaction between the jet flow at the ventilation opening and the plant distribution, the present invention improves the prediction accuracy of the image acquisition window period; The humidity-adaptive reflectivity correction of the present invention effectively overcomes the interference of water film reflection in a high-humidity environment and reduces the residual reflection amount under the condition of greenhouse environment RH>85%; At the same time, the GAN repair algorithm based on plant topological constraints of the present invention improves the structural rationality of occlusion repair and reduces the error rate by introducing prior knowledge of growth rules. The present invention improves the comprehensive pollination success rate in the tomato greenhouse scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. Among them: Figure 1 It is a schematic flow chart of the method for UAV image processing and target recognition based on deep learning of the present invention; Figure 2 It is a schematic structural diagram of the UAV image processing and target recognition system based on deep learning of the present invention; Figure 3 It is a processing flow chart of outputting a distortion-corrected image after occlusion area repair from a distortion-corrected image of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] To make the above objects, features, and advantages of the present invention more apparent and understandable, the following provides a detailed description of the specific embodiments of the present invention in conjunction with the accompanying drawings of the specification.

[0016] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, the present invention may be practiced in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0017] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation manner of the present invention. The phrase "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that mutually excludes other embodiments.

[0018] Embodiment 1: As Figure 1 shown, the method for drone image processing and target recognition based on deep learning according to the embodiment of the present invention, as Figure 1 shown, includes the following specific steps: Step 1: Collect greenhouse structure parameters, plant distribution density, and drone flight data, establish a plant-airflow coupling equation, and output the effective image window period; Step 1 includes: A11: Collect the three-dimensional model of the greenhouse structure and the plant planting layout diagram, and calibrate the greenhouse microenvironment parameters. The greenhouse microenvironment parameters include: average wind speed and plant distribution turbulence intensity value; Exemplarily, in this embodiment, the calibration strategy of the average wind speed is set based on the physical model of the jet flow at the greenhouse ventilation opening, specifically: ; Among them, represents the average wind speed at the spatial coordinates (x, y, z); is the wind speed at the greenhouse ventilation opening; is the ventilation opening coordinate, indicating the position of the ventilation opening in the greenhouse space; is the diffusion influence coefficient; Exemplarily, in this embodiment, ; is the ventilation opening area; It should be noted that the larger the ventilation opening area, the wider the airflow diffuses, and the larger the value of

[0019] It should be noted that is the height correction term. It should be noted that during the jet flow at the ventilation opening, the velocity distribution in the vertical direction is usually not uniform and changes with height.

[0020] Exemplarily, in this embodiment, the calibration strategy of the turbulent intensity value of the plant distribution is as follows: ; where H is the greenhouse height, are the row spacing data and the plant spacing data respectively; Exemplarily, in this embodiment, it should be noted that the coefficient 0.16 comes from the semi-empirical formula of turbulent flow in a closed space; where the three-dimensional model of the greenhouse structure includes: ventilation opening coordinate data; the plant planting layout diagram includes: row spacing data, plant spacing data; Exemplarily, in this embodiment, it should be noted that the calibration of greenhouse microenvironment parameters establishes an explicit correlation between the plant layout and the airflow field, providing a basis for the execution decision of subsequent work tasks; A12: Obtain the characteristic data of the plant where the target object is located and the rotor speed of the drone, calibrate the bending stiffness of the plant stem through a three-point bending dynamic experiment, and at the same time, through establishing a non-uniform force distribution model, conduct the calibration of plant mechanical parameters; The calibration strategy of the bending stiffness of the plant stem is as follows: ; where d is the stem diameter; is the concentrated force applied in the three-point bending dynamic experiment; is the support span; is the maximum mid-span deflection; The non-uniform force distribution model is specifically: ; is the distributed aerodynamic load exerted by the downwash airflow of the drone rotor on the plant stem; is the force conversion coefficient; Exemplarily, in this embodiment, it should be noted that turbulence will cause secondary disturbance to the drone rotor force. Therefore, in this embodiment, the turbulent intensity value is used to correct the attenuation coefficient of the downwash airflow of the drone rotor among plants, specifically: ; is the calibration value without turbulence; is the rotor speed of the drone; Exemplarily, in this embodiment, it should be noted that the higher the drone speed, the greater the downwash airflow velocity generated, and the greater the disturbing force on the plant; X is the longitudinal position along the plant stem; is the length of the plant stem.

[0021] Exemplarily, in this embodiment, the non-uniform force distribution model is established based on the rotor force model. It should be noted that is the spatial decay term, which reflects the linear decay of the air flow along the stem length.

[0022] Step one further includes: A21: According to A11 - A12, establish a real-time prediction equation for plant swing, and predict and obtain the displacement of the target object on the plant ; The real-time prediction equation for plant swing is specifically: ; Wherein, are the stem density and cross-sectional area respectively; c is the damping coefficient; Exemplarily, in this embodiment, the stem density is obtained by actual measurement with a microwave moisture content detector; Exemplarily, in this embodiment, it should be noted that the real-time prediction equation for plant swing is set based on the Timoshenko beam equation; In this embodiment, a method for solving the real-time prediction equation for plant swing is provided, including: Divide the stem into N segments, Δx = L / N, where the pistil is located in the k-th segment, that is ; Δx is the length of each segment when the stem is divided into N segments; Make the real-time prediction equation for plant swing dimensionless, discretize it by the spectral method, and take the first 3 modes, where the modal coefficients are solved by the Galerkin projection method; Use the central difference method to discretize the second-order spatial derivative in the control equation to obtain an approximate expression for each segment, specifically: , substitute the discretized approximate expression into the control equation to obtain the discretized control equation; Wherein, represents the values of the function y at positions i + 1, i, i - 1 in the x direction, i is the discretized position index, and Δx is the spacing between adjacent positions; Establish an equation for each segment of the stem division, solve it using the finite difference method, and finally only retain the displacement of the k-th segment as the output, and obtain the displacement at the tomato pistil as: ; Where H is the number of expansion terms, which determines how many sine functions are used to approximately represent , is the modal function, which represents the value of the h-th sine function at time t for The contribution magnitude; It should be noted that the average wind speed obtained by calculation in A11 is used as the initial condition of the real-time prediction equation of plant swing .

[0023] A22: According to the displacement of the target object on the plant and the sampling frequency of the UAV camera, calculate the real-time swing phase of the target object, determine the effective image window period, and output the effective sampling window ; The determination of the effective image window period includes: ; Among them, is the real-time swing phase of the target object.

[0024] In one specific embodiment, the calculation formula of the real-time swing phase of the target object is: ; Among them, At the position of the stamen height The speed that changes with time t; Represents the displacement at the position of the stamen height ; Exemplarily, in this embodiment, can be obtained by approximately calculating the displacement data through numerical differentiation; Step 2: Monitor the greenhouse humidity in real time, establish a humidity-reflection correlation compensation model for deformation compensation, and output the distortion-corrected image after reflection suppression; Step 2 includes: B11: Fit the relationship between the humidity reflectance correction coefficient corresponding to each band and the greenhouse humidity, use the least squares method to adjust the parameters in the relationship formula to minimize the error between the theoretical reflectance and the measured reflectance, and output the humidity-reflectance correction coefficient under the minimum error, and correct the reflectance according to the humidity-reflectance correction coefficient to obtain the corrected reflectance ; In one specific embodiment, the correction of the reflectance includes: ; Among them, are the corrected and uncorrected reflectances respectively; B12: Collect the original polarization images through a polarization camera at different polarization angles, and find the angle that minimizes the adjusted light intensity as the optimal polarization angle by traversing different polarization angles ; Exemplarily, in this embodiment, the polarization angles include: 0°, 45°, 90°, 135°; In one specific implementation, the acquisition strategy of the optimal polarization angle is: ; Wherein, is the light intensity of the original polarization image when the polarization angle is ; is the reflectivity in the current environment; B13: Through the dynamic adjustment strategy of the polarization angle, the optimal polarization angle is updated every 10 frames. The specific dynamic adjustment strategy of the polarization angle is: ; Wherein, is the polarization angle at the next moment and the previous moment; is the flower region mask; (i, j) represents a pixel point, represents the intensity value of the image at the pixel point (i, j); represents the total reflection intensity of the region; It should be noted that the flower region mask is obtained through an image segmentation algorithm to ensure that only the reflected light of the flower region is optimized.

[0025] It should be noted that the dynamic adjustment strategy of the polarization angle is based on the gradient descent algorithm. By calculating the gradient of the reflected light intensity of the flower region with respect to the polarization angle, the polarization angle is gradually adjusted step by step with a step size of 0.1 to gradually reduce the reflected light intensity of the flower region and reach the optimal state; It should be noted that in the actual greenhouse environment, humidity will change the reflectivity of the petals, thereby affecting the reflected light intensity; It should be noted that by optimizing the polarization light field, the interference of specular reflected light on the petal surface is reduced, and the image quality is improved.

[0026] B14: Extract the effective window period of the image and the angular velocity information of the drone , and calculate the best shutter trigger moment for image capture , and capture the original image at the best shutter trigger moment ; In one specific implementation, the calculation strategy of the best shutter trigger moment is: ; Wherein, is the current moment, is the current swing phase.

[0027] It should be noted that The angular velocity information of the drone is obtained in real time by using the inertial measurement unit (IMU) carried by the drone, which reflects the motion state of the drone itself and is used to determine whether motion compensation is required; It should be noted that in the greenhouse environment, the plants will swing under the influence of factors such as air flow, and the motion of the drone itself will also cause interference, which will lead to blurred images. The purpose of this step is to achieve precise triggering of the camera shutter to ensure that the images are collected at the moment when the plant swing is minimized and the drone motion is stable; Step 2 also includes: extracting the original image and the stem bending stiffness matrix, calculating the actual displacement of each control point according to the stiffness matrix and the swing displacement, and applying the reverse displacement to each pixel through TPS transformation to restore the stem to a vertical state and obtain the distortion-corrected image; In one specific embodiment, the displacement of each control point The calculation formula is: Where is the local stiffness, and Sw is the displacement caused by the plant swing; Calculate the distance from this pixel point to each control point Substitute it into the kernel function; where the kernel function is and is used to describe the influence degree of the control point on the surrounding pixels; In one specific embodiment, the acquisition strategy of the corrected image is: ; is the corrected image; (x, y) is the image pixel coordinate, is the coordinate of the control point set along the stem, with a total of N, and n is the index number of the control point. A control point is set every 5 cm along the stem; is the weight coefficient, which is solved by stiffness constraint, and the weight will be automatically allocated according to the control point displacement and stiffness, and the weight of the control point closer to the target is larger; Exemplarily, in this embodiment, a strategy for obtaining a weight coefficient is given, which specifically includes: establishing an objective function , and minimizing the objective function to obtain the weight coefficient ; Where The term represents the constraint of the stem stiffness on the deformation to ensure that the deformation conforms to the mechanical properties of the plant; The term is the Laplacian smoothing term, is a balance parameter used to adjust the relationship between stiffness constraints and smoothing effects. L is the Laplacian smoothing operator, ensuring that the deformed image is smooth and natural.

[0028] It should be noted that the swaying of the plant will cause non-rigid deformation of the image, affecting the accuracy of subsequent image analysis. By establishing a deformation compensation model based on the stem stiffness, the distortion correction of the image can be completed.

[0029] Step 3: Based on the distortion-corrected image, through the GAN repair under topological constraints, process the area occluded by the plant in the image, and output the candidate map of the target object after the occlusion area is repaired; As Figure 3 shown, Step 3 includes: C11: Obtain the row spacing data and layer height data of the plant planting. At the same time, collect the three-dimensional coordinate information of each point of the plant through the RGB-D camera. Remove the noise points through preprocessing, retain the effective depth point cloud data, and construct the plant topological skeleton map according to the effective depth point cloud data; It should be noted that the row spacing data is used to determine the search radius for main stem positioning, and the layer height data serves as the vertical spacing benchmark for lateral branch generation; Exemplarily, in this embodiment, a method for extracting the plant topological skeleton is provided. For tomato plants, the density clustering method is used to locate the position of the main stem. At the same time, the layered growth pattern of the plant lateral branches is simulated, and the generation of lateral branches is simulated around the position of the main stem. It should be noted that tomato plants usually arrange their lateral branches in a spiral shape, and the growth direction of each layer of lateral branches has periodicity.

[0030] Exemplarily, in this embodiment, it should be noted that the plant topological skeleton map is used to ensure that the repaired area conforms to the plant growth law; C12: Obtain the plant topological skeleton map and use the generative adversarial network (GAN) for occlusion repair. The occlusion repair includes: Concatenate the noise vector and the plant topological skeleton map in the channel dimension as the input of the generator; Among them, the noise vector is a 100-dimensional random vector; the plant topological skeleton map is (a grayscale image, the skeleton area is white, and the background is black) Through the topological attention module, the convolutional layer generates a query vector Q from the skeleton map to capture the structural features, and the convolutional layer generates a key vector K and a value vector V from the occluded image to focus on the visual features of the occluded area, enabling the generator to repair the occluded area under the guidance of the skeleton; Exemplarily, in this embodiment, the generator repairing the occluded area under the guidance of the skeleton includes: generating reasonable leaf or flower and fruit forms according to the intersection structure of the main stem and lateral branches; At the discriminator input, the real image and the repaired image are input. The pre-trained U-Net model SkelNet is used to extract the skeleton, and the topological consistency loss is calculated through the L1 norm. Combined with the VGG network feature extraction layer, the perceptual loss is calculated through the L2 norm. ; Calculate the total loss LOSS. The calculation strategy of the total loss is as follows: ; Among them, is the adversarial loss; It should be noted that the adversarial loss prompts the generator to generate realistic images; Exemplarily, in this embodiment, the training adopts an asymmetric strategy. In the initial stage, the weight of the total loss is increased to ensure the correct structure, and in the later stage, the weight is gradually reduced to optimize the visual details, so as to achieve occlusion repair that conforms to the growth law of plants.

[0031] C13: Obtain the image after occlusion repair.

[0032] Step three further includes: C21: Convert the original image and the repaired image to the LAB color space and calculate the color difference value. The calculation strategy of the color difference value is as follows: ; Among them, is the original image; is the image after occlusion repair; is the color space conversion function; is the L2 norm; It should be noted that the color space conversion function is used to convert the input image from a color space such as RGB to the CIELAB color space; It should be noted that the L2 norm is used to calculate the square root of the sum of the squares of the corresponding elements of two vectors C22: Preset the color difference standard threshold, traverse all the repaired areas in the repaired image. When the color difference value of the current area is less than the color difference standard threshold, it is determined that the current area is a reliable repaired area with consistent illumination, and the repair is retained; When the color difference value of the current area is greater than or equal to the color difference standard threshold, it is determined that the current area is an unreliable repaired area with inconsistent illumination, and the repair is revoked; C23: Output the candidate map of the target object after occlusion area repair.

[0033] Step four: Evaluate the state of the target object to provide a quantitative index for the work task decision.

[0034] Exemplarily, in this embodiment, the target object is a tomato stamen. In this embodiment, a specific implementation manner of step four is given, including: D11: Extract the repaired image output by step three and the environmental temperature and environmental humidity data RH in the greenhouse; D12: Perform a state assessment on the tomato flower stamens, including: calculating the flower stamen wilting index according to the results of the opening angle assessment and the saturation assessment; The opening angle assessment includes: using the Sobel operator to perform target object edge detection on the gradient image to obtain a single-pixel edge, with the flower stamen center (calculated by the centroid algorithm) as the origin, converting the edge points to polar coordinates, and calculating the average opening angle: ; Among them, is the coordinate of the m-th effective edge point; M is the number of effective edge points, and m is the label index of the effective edge points; The saturation assessment includes: converting the RGB image to the HSV space, calculating the flower stamen saturation value, and simultaneously correcting the flower stamen saturation according to the greenhouse humidity data to obtain the corrected flower stamen saturation value; Embodiment 2: As Figure 2 shown, the deep learning-based UAV image processing and target recognition system according to the embodiment of the present invention, as Figure 2 shown, includes the following modules: Window period determination module, distortion correction module, occlusion repair module, task decision module, and main control module; The window period determination module is used to collect greenhouse structure parameters, plant distribution density, and UAV flight data, establish a plant-airflow coupling equation, and output an effective image window period; The distortion correction module is used to monitor the greenhouse humidity in real time, establish a humidity-reflection correlation compensation model for deformation compensation, and output a distortion-corrected image after reflection suppression; The occlusion repair module is used to process the area occluded by plants in the image based on the distortion-corrected image through GAN repair under topological constraints, and output a target object candidate map after repairing the occluded area; The task decision module is used to evaluate the state of the target object and provide a quantitative index for work task decision-making; The main control module is used to run and control other modules.

[0035] Embodiment 3: This embodiment provides an electronic device, including: a processor and a memory, where a computer program that can be called by the processor is stored in the memory; The processor executes the above-mentioned deep learning-based UAV image processing and target recognition method by calling the computer program stored in the memory.

[0036] The electronic device may vary greatly due to different configurations or performances, and can include one or more processors (Central Processing Units, CPU) and one or more memories. Among them, at least one computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the deep learning-based UAV image processing and target recognition method provided by the above method embodiment. The electronic device can also include other components for realizing the functions of the device. For example, the electronic device can also have components such as wired or wireless network interfaces and input / output interfaces for data input and output. This embodiment will not be elaborated here.

[0037] Embodiment 4: This embodiment proposes a computer-readable storage medium, on which a rewritable computer program is stored; When the computer program runs on a computer device, it causes the computer device to execute the above-mentioned deep learning-based UAV image processing and target recognition method.

[0038] For example, the computer-readable storage medium can be a read-only memory (Read-Only Memory, abbreviated as ROM), a random access memory (Random Access Memory, abbreviated as RAM), a compact disc read-only memory (Compact Disc Read-Only Memory, abbreviated as CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0039] It should be understood that in various embodiments of the present application, the magnitudes of the serial numbers of the above processes do not mean the order of execution is prior or posterior. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0040] It should be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0041] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions according to the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired network or / and a wireless network. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that contains a collection of one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0042] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0043] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification only illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and all these changes and improvements fall within the scope of the present invention claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for processing drone images and target recognition based on deep learning, characterized in that, The method includes: Step 1: Collect greenhouse structure parameters, plant distribution density, and UAV flight data, establish a plant-airflow coupling equation, and output the effective image window period; Step 2: Monitor the greenhouse humidity in real time, establish a humidity-reflection correlation compensation model for deformation compensation, and output the distortion-corrected image after reflection suppression; Step 3: Based on the distortion-corrected image, perform processing on the areas occluded by plants in the image through GAN repair under topological constraints, and output the candidate map of the target object after occlusion area repair; Step 4: Evaluate the state of the target object and provide quantitative indicators for work task decision-making.

2. The method for processing drone images and target recognition based on deep learning according to claim 1, wherein, Step 1 includes: A11: Collect the three-dimensional model of the greenhouse structure and the plant planting layout map, and calibrate the greenhouse microenvironment parameters. The greenhouse microenvironment parameters include: average wind speed and plant distribution turbulence intensity value; Among them, the three-dimensional model of the greenhouse structure includes: ventilation port coordinate data; the plant planting layout map includes: row spacing data and plant spacing data; A12: Obtain the characteristic data of the plant where the target object is located and the rotor speed of the UAV, calibrate the bending stiffness of the plant stem through a three-point bending dynamic experiment, and at the same time, perform plant mechanical parameter calibration by establishing a non-uniform force distribution model; The bending stiffness of the stalk The calibration strategy is as follows: ; Among them, d is the stem diameter; is the concentrated force applied in the three-point bending dynamic experiment; is the support span; is the maximum mid-span deflection; The non-uniform force distribution model is specifically as follows: ; is the distributed aerodynamic load exerted by the downwash airflow of the drone rotor on the plant stem; is the force conversion coefficient; is the rotor speed of the drone; X is the longitudinal position along the plant stem; is the length of the plant stem.

3. The method for processing drone images and target recognition based on deep learning according to claim 2, characterized in that, Step 1 also includes: A21: Based on A11 - A12, establish a real-time prediction equation for plant swaying to predict and obtain the displacement of the target object on the plant ; The specific plant swing real-time prediction equation is: ； Among them, are the stalk density and cross-sectional area respectively; c is the damping coefficient; A22: Based on the displacement of the target on the plant and the sampling frequency of the UAV camera, calculate the real-time swing phase of the target, determine the effective window period of the image, and output the effective sampling window ; The determination of the effective image window period includes: ； Among them, is the real-time swing phase of the target object.

4. The method for processing drone images and target recognition based on deep learning according to claim 3, characterized in that Step 2 includes: B11: Fit the relationship between the humidity reflectance correction coefficient corresponding to each band and the greenhouse humidity. Using the least squares method, adjust the parameters in the relationship formula to minimize the error between the theoretical reflectance and the measured reflectance, and output the humidity-reflectance correction coefficient under the minimum error , and according to the humidity-reflectance correction coefficient correct the reflectance to obtain the corrected reflectance ; B12: The original polarization images are acquired by a polarization camera at different polarization angles. By traversing different polarization angles, the angle that minimizes the adjusted light intensity is found as the optimal polarization angle ; B13: Through the dynamic adjustment strategy of the polarization angle, update the optimal polarization angle every 10 frames. The specific dynamic adjustment strategy of the polarization angle is: ； Among them, is the polarization angle at the next moment and the previous moment; is the flower region mask; (i,j) represents a pixel point, represents the intensity value of the image at the pixel point (i,j); Represents the total reflection intensity of the region; B14: Extract the effective window period of the image and the angular velocity information of the drone , the optimal shutter trigger moment for image capture , and at the optimal shutter trigger moment capture the original image.

5. The method for processing drone images and target recognition based on deep learning according to claim 4, characterized in that, Step 2 also includes: Extract the original image and the stem bending stiffness matrix, calculate the actual displacement of each control point according to the stiffness matrix and the swing displacement, and apply the reverse displacement to each pixel through TPS transformation to make the stem return to the vertical state and obtain the distortion-corrected image.

6. The method for processing drone images and target recognition based on deep learning according to claim 5, wherein, Step 3 includes: C11: Obtain the row spacing data and floor height data of plant planting, and at the same time collect the three-dimensional coordinate information of each point of the plant. Remove the noise points through preprocessing, retain the effective depth point cloud data, and construct a plant topological skeleton map according to the effective depth point cloud data; C12: Obtain the plant topological skeleton map and use the generative adversarial network for occlusion repair. The occlusion repair includes: Concatenate the noise vector and the plant topological skeleton map in the channel dimension as the input of the generator; Through the topological attention module, from the convolutional layer generate a query vector Q from the skeleton graph to capture structural features, and the convolutional layer generate a key vector K and a value vector V from the occluded image to focus on the visual features of the occluded area, enabling the generator to repair the occluded area under the guidance of the skeleton; At the discriminator input, the real image and the restored image are input. The pre-trained U-Net model SkelNet is used to extract the skeleton, and the topological consistency loss is calculated through the L1 norm , combined with the VGG network feature extraction layer, the perceptual loss is calculated through the L2 norm ; Calculate the total loss LOSS. The calculation strategy of the total loss is: ； Among them, is the adversarial loss; C13: Obtain the image after occlusion repair.

7. The method for processing drone images and target recognition based on deep learning according to claim 6, characterized in that, Step 3 also includes: C21: Convert the original image and the restored image to the LAB color space and calculate the color difference value. The calculation strategy for the color difference value is as follows: ； Among them, is the original image; is the image after occlusion repair; is a color space conversion function; is the L2 norm; C22: Preset the color difference standard threshold, traverse all the repaired areas in the repaired image. When the color difference value of the current area is less than the color difference standard threshold, determine that the current area is a reliable repaired area with consistent illumination and retain the repair; When the color difference value of the current area is greater than or equal to the color difference standard threshold, determine that the current area is an unreliable repaired area with inconsistent illumination and revoke the repair; C23: Output the candidate map of the target object after occlusion area repair.

8. A system for an image processing and target recognition method of an unmanned aerial vehicle based on deep learning, which is used to implement the image processing and target recognition method of an unmanned aerial vehicle based on deep learning according to any one of claims 1-7, characterized in that, The system includes: Window period determination module, distortion correction module, occlusion repair module, task decision-making module, and main control module; The window period determination module is used to collect greenhouse structure parameters, plant distribution density, and UAV flight data, establish a plant-airflow coupling equation, and output the effective image window period; The distortion correction module is used to monitor the greenhouse humidity in real time, establish a humidity-reflection correlation compensation model for deformation compensation, and output a distortion-corrected image after reflection suppression; The occlusion repair module is used to process the areas occluded by plants in the image based on the distortion-corrected image through GAN repair under topological constraints, and output a candidate map of the target object after the occlusion area is repaired; The task decision module is used to evaluate the state of the target object and provide quantitative indicators for work task decision-making; The main control module is used to run and control other modules.

Images