Image target detection method, system and electronic equipment based on drone

Abstract

The present application relates to the field of target detection, and specifically discloses an image target detection method, system and electronic device based on a drone, which first obtains a panoramic image of a farmland collected by a high-resolution camera and an infrared video of the farmland obtained by an infrared camera, then passes the panoramic image of the farmland through multiple convolutional layers, and then obtains a panoramic farmland channel feature vector by using a convolutional neural network model of a target detection network and performing permutation and global eigenvalue mean pooling operations, then extracts key frames from the infrared video of the farmland and passes them through a spatial attention feature extractor based on a convolutional neural network to obtain an infrared image feature vector, and finally, fuses the panoramic farmland channel feature vector and the infrared image feature vector and uses a classifier to determine whether to issue a crop pest warning prompt, so as to improve the effect of farmland pest monitoring, thereby providing more accurate warning prompts for crop protection and management, and thus helping to take timely measures to avoid losses.

Description

Technical Field

[0001] The present application relates to the field of target detection, and more specifically, to a method, system and electronic device for detecting image targets based on a drone. Background Art

[0002] Crop pests refer to the damage caused to crops by pests that appear in large numbers in farmland. Pests can directly eat the leaves, stems and fruits of plants, and can also spread viruses and bacteria, resulting in stunted growth, reduced yields and even death of crops. Crop pests are the number one natural biological disaster that threatens crop yield and quality, and are easily affected by many factors such as crop variety, crop planting method and growth environment.

[0003] Crop pests have caused serious economic losses to agricultural production. Farmers have adopted methods such as patrolling farmland to deal with pests. However, farmland is vast. If pest identification relies on manual patrols, it will take a lot of time and energy. Manual identification is inefficient and cannot quickly cover large areas of farmland, resulting in delayed and inaccurate pest monitoring. In addition, manual identification is limited by the human eye's field of vision and observation angle, making it difficult to fully observe and analyze the distribution and extent of farmland pests. It is also unable to capture tiny pest characteristics, resulting in missed detection and misjudgment of pests.

[0004] Therefore, a method, system and electronic device for detecting image targets based on a drone are desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides an image target detection method, system and electronic device based on a drone, which first obtains a panoramic image of a farmland collected by a high-resolution camera and an infrared video of a farmland obtained by an infrared camera, and then obtains a panoramic farmland channel feature vector after the panoramic image of the farmland passes through a multi-layer convolution layer by using a convolutional neural network model of a target detection network and performing arrangement and global eigenvalue mean pooling operations, then extracts key frames from the farmland infrared video and passes through a spatial attention feature extractor based on a convolutional neural network to obtain an infrared image feature vector, and finally, fuses the panoramic farmland channel feature vector and the infrared image feature vector and uses a classifier to determine whether to issue a crop pest warning prompt, so as to improve the effect of farmland pest monitoring, thereby providing more accurate warning prompts for crop protection and management, and thus helping to take timely measures to avoid losses.

[0006] According to one aspect of the present application, a method for detecting an image target based on a drone is provided, which comprises:

[0007] Obtain panoramic images of farmland collected by high-resolution cameras and infrared videos of farmland obtained by infrared cameras;

[0008] Extracting a panoramic farmland channel feature vector and an infrared image feature vector from the farmland panoramic image and the farmland infrared video;

[0009] Constructing a farmland classification feature vector between the panoramic farmland channel feature vector and the infrared image feature vector;

[0010] Based on the farmland classification feature vector, it is determined whether to issue a crop pest warning prompt.

[0011] The above-mentioned drone-based image target detection method also includes a training method for training the multi-layer convolutional layers, the convolutional neural network model as the target detection network, the spatial attention mechanism using the convolutional neural network, and the classifier.

[0012] In the above-mentioned drone-based image target detection method, the training method for training the multi-layer convolutional layers, the convolutional neural network model as the target detection network, the spatial attention mechanism using the convolutional neural network and the classifier includes: obtaining training data, the training data including a training farmland panoramic image acquired by a high-resolution camera and a training farmland infrared video acquired by an infrared camera; passing the training farmland panoramic image through a multi-layer convolutional layer to obtain a training panoramic image feature map; passing the training panoramic image feature map through a convolutional neural network model using a target detection network to obtain a plurality of training farmland regions of interest; arranging the plurality of training farmland regions of interest according to sample dimensions into a training farmland region feature tensor; performing global eigenvalue mean pooling on the farmland regions of interest at each position in the training farmland region feature tensor to obtain the training panoramic farmland channel feature vector; Extract key frames from the training farmland infrared video to obtain multiple training infrared key frames; pass the multiple training infrared key frames through a spatial attention feature extractor based on a convolutional neural network to obtain the training infrared image feature vector; construct a training farmland classification feature vector between the training panoramic farmland channel feature vector and the training infrared image feature vector; calculate a geometric non-rigid consistency factor based on order prior between the training panoramic farmland channel feature vector and the training infrared image feature vector; pass the training farmland classification feature vector through a classifier to obtain a classification loss function value; use the weighted sum between the classification loss function value and the geometric non-rigid consistency factor based on order prior as the loss function value, and train the multi-layer convolutional layer, the convolutional neural network model as a target detection network, the spatial attention mechanism using a convolutional neural network, and the classifier.

[0013] In the above-mentioned drone-based image target detection method, calculating the geometric non-rigid consistency factor based on order prior between the training panoramic farmland channel feature vector and the training infrared image feature vector includes: calculating the geometric non-rigid consistency factor based on order prior between the training panoramic farmland channel feature vector and the training infrared image feature vector using the following geometric non-rigid consistency factor formula;

[0014] The geometric non-rigid uniformization factor formula is:

[0015]

[0016] Among them, V1 represents the training panoramic farmland channel feature vector, V2 represents the training infrared image feature vector, ||·||2 represents the bi-norm of the feature vector, cos(V1, V2) represents the calculation of the cosine distance between the training panoramic farmland channel feature vector and the training infrared image feature vector, and Loss represents the geometric non-rigid unification factor based on order prior.

[0017] According to another aspect of the present application, there is provided an image target detection system based on a drone, comprising:

[0018] The farmland visual data acquisition module is used to obtain the farmland panoramic images acquired by the high-resolution camera and the farmland infrared video acquired by the infrared camera;

[0019] A farmland visual feature extraction module is used to extract a panoramic farmland channel feature vector and an infrared image feature vector from the farmland panoramic image and the farmland infrared video;

[0020] A construction module, used to construct a farmland classification feature vector between the panoramic farmland channel feature vector and the infrared image feature vector;

[0021] The pest result generation module is used to determine whether to issue a crop pest warning prompt based on the farmland classification feature vector.

[0022] According to another aspect of the present application, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program.

[0023] Compared with the prior art, the present application provides a method, system and electronic device for image target detection based on a drone, which first obtains a panoramic image of a farmland captured by a high-resolution camera and an infrared video of a farmland captured by an infrared camera, and then obtains a panoramic farmland channel feature vector after passing the panoramic image of the farmland through multiple convolutional layers using a convolutional neural network model of a target detection network and performing arrangements and global eigenvalue mean pooling operations. Next, the key frames of the farmland infrared video are extracted and passed through a spatial attention feature extractor based on a convolutional neural network to obtain an infrared image feature vector. Finally, the panoramic farmland channel feature vector and the infrared image feature vector are fused and a classifier is used to determine whether to issue a crop pest warning prompt, so as to improve the effect of farmland pest monitoring, thereby providing more accurate warning prompts for crop protection and management, and thus helping to take timely measures to avoid losses. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0025] Figure 1 Flow chart of a method for detecting target images based on a drone according to an embodiment of the present application.

[0026] Figure 2 is a block diagram of an image target detection system based on a drone according to an embodiment of the present application.

[0027] Figure 3 Schematic diagram of the architecture of an image target detection system based on a drone according to an embodiment of the present application.

[0028] Figure 4 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0030] Exemplary Methods

[0031] Figure 1 FIG. 1 is a flow chart of a method for detecting an image target based on a drone according to an embodiment of the present application. Figure 1As shown, according to the UAV-based image target detection method of the embodiment of the present application, it includes: S110, acquiring a panoramic image of a farmland captured by a high-resolution camera and an infrared video of a farmland acquired by an infrared camera; S120, extracting a panoramic farmland channel feature vector and an infrared image feature vector from the panoramic image of the farmland and the infrared video of the farmland; S130, constructing a farmland classification feature vector between the panoramic farmland channel feature vector and the infrared image feature vector; S140, judging whether to issue a crop pest warning prompt based on the farmland classification feature vector.

[0032] In step S110, a panoramic image of a farmland captured by a high-resolution camera and an infrared video of a farmland captured by an infrared camera are obtained. It should be understood that a high-resolution camera is a camera with a higher pixel density, which can capture more details and more precise images. The resolution of a camera refers to the number of pixels in an image, usually expressed in the form of horizontal pixels and vertical pixels. A high-resolution camera has more pixels and can produce larger and clearer images. A higher pixel density means that more pixels are used to represent an image, thereby providing more information and better image quality. A high-resolution camera can capture smaller details, richer colors, and smoother tonal transitions, making the image more realistic and delicate. In addition, a high-resolution camera also has a greater image magnification capability, which allows for more cropping and magnification in post-processing without losing too much detail. An infrared camera is an advanced optical device that can capture and record infrared radiation and convert it into a visible image or thermal map. The working principle of an infrared camera is based on the physical properties of infrared radiation, using infrared sensors and appropriate optical systems to capture and convert infrared radiation into a visual image. Compared with traditional visible light cameras, infrared cameras can work in complete darkness or low-light environments and can detect thermal radiation that is imperceptible to the human eye. Infrared cameras are used in the agricultural field. They can help farmers find and deal with pest problems in a timely manner by detecting temperature changes caused by pests. Pests usually cause local temperature increases in infected plants, which is due to the heat generated by the metabolic process caused by pest activity. The advantage of using infrared cameras for pest monitoring is that it can scan quickly over a large area and does not require direct contact with plants, which reduces interference with crops and improves the efficiency and accuracy of detection.

[0033] Panoramic images can provide an overall view of the farmland, including information such as crop distribution and leaf status. By observing the color, shape and damage of crop leaves in the image, it is possible to preliminarily determine whether there are pests and help determine the distribution range and hotspot areas of pests. Considering that pests usually cause leaf color changes, leaf damage or morphological deformities, the presence and extent of pests can be identified by comparing normal leaves with leaves affected by pests. At the same time, panoramic images of farmland can capture the traces and damage left by pests. For example, some pests will leave bite marks, holes or feces on the leaves. By observing these traces and damage in the image, the type and extent of pests can be determined. Infrared cameras can capture infrared radiation from farmland, which is very useful for detecting the health status and pest and disease conditions of plants in farmland. When plants are attacked by pests and diseases, they will produce heat distribution and infrared radiation characteristics that are different from normal plants. Through infrared video, these abnormalities can be detected, thereby providing early warning of crop pests. By combining the panoramic images captured by high-resolution cameras and the infrared videos acquired by infrared cameras, more comprehensive farmland information can be obtained. The panoramic images provide image features within the visible spectrum, while the infrared videos provide the heat distribution and infrared radiation characteristics of the plants. The combination of the two can more accurately judge the health status and pest situation of crops.

[0034] In step S120, a panoramic farmland channel feature vector and an infrared image feature vector are extracted from the farmland panoramic image and the farmland infrared video. It should be understood that the panoramic farmland channel feature vector is a feature vector extracted from the farmland panoramic image, which may contain information of various positions in the farmland, such as vegetation distribution, soil conditions, etc., which are important for judging the growth status of crops and possible pests. By extracting the panoramic farmland channel feature vector, the overall characteristics of the farmland can be obtained to help identify potential pest areas. The infrared image feature vector is a feature vector extracted from the farmland infrared video. The infrared image can provide information different from the visible light image. Pests usually cause temperature changes in plant tissues. The infrared image can capture these temperature changes, thereby providing potential clues to pests. By extracting the infrared image feature vector, the infrared information can be used to assist in judging whether crops are affected by pests. By comprehensively utilizing the panoramic farmland channel feature vector and the infrared image feature vector, more comprehensive and multi-dimensional farmland information can be obtained, thereby improving the accuracy and reliability of crop pest warning.

[0035] In a specific embodiment of the present application, the step S120, performing feature extraction on the farmland panoramic image and the farmland infrared video to obtain a panoramic farmland channel feature vector and an infrared image feature vector, includes: S121, passing the farmland panoramic image through multiple convolution layers to obtain a panoramic image feature map; S122, performing feature extraction on the panoramic image feature map to obtain the panoramic farmland channel feature vector; S123, performing feature extraction on the farmland infrared video to obtain the infrared image feature vector. In order to extract feature information from the farmland panoramic image, the farmland panoramic image is passed through multiple convolution layers to obtain a panoramic image feature map. It should be understood that the convolution layer is a network layer commonly used in deep learning. It can perform convolution operations on the input image through filters (also called convolution kernels) to extract local features of the image. The superposition of multiple convolution layers can gradually extract higher-level features of the image, from edges and textures to more complex shapes and structures. In the specific application scenario of farmland pest prevention, since different regions may have different vegetation, soil, pest and other characteristics, by inputting the panoramic image into the multi-layer convolution layer, the convolution operation can be used to capture the local features in the image. Each convolution layer will filter the input image and extract the features that match the filter. As the number of layers increases, the network can learn more abstract and higher-level features, which can better represent the content in the image. Specifically, each layer of the multi-layer convolution layer performs convolution processing, pooling processing and activation processing on the input data in the forward pass of the layer to generate the panoramic image feature map by the last layer of the multi-layer convolution layer, wherein the input of the first layer of the multi-layer convolution layer is the farmland panoramic image.

[0036] Furthermore, by obtaining the panoramic image feature map, the farmland panoramic image can be converted into a series of feature vectors, each of which corresponds to a local area in the image. It should be understood that feature extraction of the panoramic image feature map can convert the information in the image into a numerical vector that can be used for classification and prediction. As mentioned above, the panoramic image feature map is obtained through multiple layers of convolutional layers, which contains feature representations of various positions in the image. However, the dimensions of these feature maps are usually very high. Therefore, it is necessary to further extract features from the feature map to convert it into a more compact and representative feature vector. Specifically, feature extraction can be achieved through various methods, such as using pooling operations, dimensionality reduction algorithms (such as principal component analysis, linear discriminant analysis, etc.), feature selection, etc. These methods can extract the most representative features from the feature map, while reducing the dimension of the features, making the feature vector easier to process and analyze. The panoramic farmland channel feature vector is obtained by extracting features from the panoramic image feature map. It contains representative features of each location in the farmland. These feature vectors can be used to describe the overall characteristics of the farmland, such as vegetation distribution, soil conditions, etc. By extracting the panoramic farmland channel feature vector, the complex information in the original image can be converted into a concise and useful numerical representation, providing input for subsequent pest classification and early warning.

[0037] Furthermore, by performing feature extraction on infrared videos of farmland, the infrared information in the video can be converted into a numerical vector, in which each element represents a feature. These feature vectors can be used to describe the characteristics of infrared images of farmland, such as vegetation conditions, thermal conditions, etc. By analyzing and processing the feature vectors of infrared images, information about the conditions of farmland and possible pests can be obtained.

[0038] In a specific embodiment of the present application, the step S122, extracting features from the panoramic image feature map to obtain the panoramic farmland channel feature vector, includes: S1221, extracting multiple farmland regions of interest from the panoramic image feature map by using a convolutional neural network model of a target detection network; S1222, reducing the dimensions of the multiple farmland regions of interest to obtain the panoramic farmland channel feature vector. It should be understood that dividing the panoramic image into multiple farmland regions of interest can be analyzed in a more fine-grained manner. Considering that each area may have different pest conditions, by analyzing each area, the distribution and degree of pests can be understood, which helps farmers determine the hot spots of pests and adopt targeted prevention and control strategies. Through the target detection network, pest targets in the panoramic image can be detected and located. These targets may be signs of pests, crops or other pests. By accurately detecting pest targets, the presence of pests can be detected early, helping farmers take corresponding prevention and control measures. Specifically, in a specific embodiment of the present application, the convolutional neural network model using the target detection network is CenterNet, ExtremeNet or RepPoints.

[0039] It should be understood that the target detection network is a computer vision technology that aims to accurately detect and locate multiple targets from images or videos. It is a neural network model in deep learning. By learning a large amount of labeled training data, it can automatically learn the characteristics of the target and perform target detection and positioning in unknown images. The target detection network usually consists of two main components: a feature extraction network and a target localization network. The feature extraction network is responsible for extracting feature representations of relevant targets from the input image, usually using a convolutional neural network to capture local and global features in the image, and the target localization network uses these feature representations to predict the location and category of the target. In the technical solution of the present application, the target detection formula is:

[0040] ROI=H(ψ det ,B)=(cls(ψ det ,B),Regr(ψ det ,B))

[0041] Among them, ψ det is the panoramic image feature map, B is the anchor box, ROI is the region of interest corresponding to the farmland, cls(ψ det ,B) indicates classification, Regr(ψ det ,B) indicates regression.

[0042] Furthermore, in order to reduce the dimension of the data and extract more representative features, multiple farmland regions of interest are reduced in dimension to obtain panoramic farmland channel feature vectors. It should be understood that farmland regions of interest may contain a large amount of pixel information. If the original data is directly used for analysis, it will lead to a huge amount of data and increase the burden of calculation and storage. Through dimensionality reduction, the high-dimensional farmland region of interest data can be compressed into a low-dimensional space, reducing the amount of data and improving calculation efficiency. At the same time, dimensionality reduction can help extract more representative features. In the farmland region of interest, there may be a large amount of redundant and irrelevant information. Through dimensionality reduction, this redundant information can be removed and the most useful features for pest monitoring and analysis can be retained. Dimensionality reduction can improve the robustness and discrimination of features, which helps to more accurately distinguish different farmland regions of interest.

[0043] In a specific embodiment of the present application, the step S1222, the multiple farmland regions of interest are reduced in dimension to obtain the panoramic farmland channel feature vector, including: arranging the multiple farmland regions of interest according to the sample dimension into a farmland region feature tensor; performing global eigenvalue mean pooling on the farmland regions of interest at each position in the farmland region feature tensor to obtain the panoramic farmland channel feature vector. It should be understood that by arranging the regional features according to the samples, a unified data structure can be formed so that each sample has the same feature dimension and shape, so that the data structure is unified. Then, the dimension of the farmland region feature tensor is reduced to a fixed feature vector by global eigenvalue mean pooling to reduce the dimension of the data, simplify the complexity of the model and calculation, and improve the calculation efficiency and reasoning speed. It should be understood that mean pooling averages the eigenvalues ​​in each region. Through the mean pooling operation, the important feature information in each farmland region of interest can be retained, which is equivalent to a summary and induction of the features in the region, and the main feature trends and distribution in the region are retained. In addition, through global eigenvalue mean pooling, the feature information at the regional level can be converted into the feature representation at the overall level to better reflect the overall state and characteristics of the farmland. Specifically, the feature matrices of the farmland regional feature tensor along the channel dimension are subjected to global mean pooling to obtain the panoramic farmland channel feature vector.

[0044] In a specific embodiment of the present application, the step S123, extracting features from the farmland infrared video to obtain the infrared image feature vector, includes: extracting key frames from the farmland infrared video to obtain multiple infrared key frames; passing the multiple infrared key frames through a spatial attention feature extractor based on a convolutional neural network to obtain the infrared image feature vector. It should be understood that the farmland infrared video contains a large number of farmland image frames with a high amount of data, but not every frame contains key information, and also contains a large amount of redundant information. By extracting key frames, frames with important information in the video can be screened out, such as frames with pests, diseases or other abnormal conditions in the farmland. This can more effectively concentrate attention and resources for subsequent analysis and processing. At the same time, while maintaining important information, it can reduce unnecessary redundant data and improve data utilization efficiency.

[0045] Furthermore, by inputting the infrared keyframes into the spatial attention feature extractor model based on convolutional neural networks, a set of feature vectors can be extracted from each keyframe. These feature vectors can better represent the content and features of the infrared image and capture information related to crop pests. It should be understood that the spatial attention feature extractor based on convolutional neural networks can help the model pay more attention to important areas and features in infrared images. By introducing the spatial attention mechanism, the network can automatically learn the importance weights of different areas in the image and pay more attention to areas related to crop pests. This helps to improve the model's ability to capture key features related to pests in infrared images. In this way, while gradually converting low-level features in the image into more semantic and abstract high-level features, the feature vectors of multiple keyframes can be integrated and fused to obtain a more comprehensive and integrated infrared image feature vector. Specifically, each layer of the spatial attention feature extractor based on the convolutional neural network performs the following on the input data in the forward pass of the layer: convolution kernel-based convolution processing on the input data to generate an infrared convolution feature map; pooling processing on the infrared convolution feature map to generate an infrared pooling feature map; activation processing on the infrared pooling feature map to generate an infrared activation feature map; global average pooling along the channel dimension on the infrared activation feature map to obtain an infrared spatial feature matrix; convolution processing and activation processing on the infrared spatial feature matrix to generate an infrared weight vector; weighting each feature matrix of the infrared activation feature map with the weight value of each position in the infrared weight vector to obtain an infrared generation feature map; pooling processing on the infrared generation feature map to obtain an infrared image feature vector.

[0046] In step S130, a farmland classification feature vector is constructed between the panoramic farmland channel feature vector and the infrared image feature vector. It should be understood that the panoramic farmland channel feature vector and the infrared image feature vector capture different farmland information respectively. The panoramic farmland channel feature vector is extracted from multiple key frames and can reflect the overall characteristics and status of the farmland. The infrared image feature vector is more focused on the farmland characteristics and pest information in the infrared image. By fusing these two features, the information of different features can be comprehensively utilized to more comprehensively describe the situation of the farmland.

[0047] In a specific embodiment of the present application, the step S130 of constructing a farmland classification feature vector between the panoramic farmland channel feature vector and the infrared image feature vector includes:

[0048] The panoramic farmland channel feature vector and the infrared image feature vector are fused using a cascade function to obtain the farmland classification feature vector, wherein the cascade function is expressed by a formula:

[0049] f(X i ,X j )=Relu(W f [θ(X i ),φ(X j )])

[0050] Among them, W f ,θ(X i ) and φ(X j ) indicates point convolution of the input, Relu is the activation function, [] indicates concatenation operation, X i represents the eigenvalue of each position in the panoramic farmland channel eigenvector, X j Represents the eigenvalue of each position in the infrared image feature vector.

[0051] In step S140, based on the farmland classification feature vector, it is determined whether to issue a crop pest warning prompt. In a specific embodiment of the present application, the step S140, based on the farmland classification feature vector, determines whether to issue a crop pest warning prompt, including: passing the farmland classification feature vector through a classifier to obtain a classification result, and the classification result is used to determine whether to issue a crop pest warning prompt. Among them, the classification formula is: softmax{(W n ,B n ):…:(W1,B1)|V}, where W1 to W n is the weight matrix, B1 to B n is the bias vector, V is the farmland classification feature vector, softmax represents the normalized exponential function, and O represents the classification result.

[0052] In a specific embodiment of the present application, it also includes a training method for training the multi-layer convolutional layer, the convolutional neural network model as the target detection network, the spatial attention mechanism using the convolutional neural network, and the classifier.

[0053] In a specific embodiment of the present application, a training method for training the multi-layer convolutional layer, the convolutional neural network model as a target detection network, the spatial attention mechanism using the convolutional neural network, and the classifier includes: obtaining training data, the training data including a training farmland panoramic image acquired by a high-resolution camera and a training farmland infrared video acquired by an infrared camera; passing the training farmland panoramic image through a multi-layer convolutional layer to obtain a training panoramic image feature map; passing the training panoramic image feature map through a convolutional neural network model using a target detection network to obtain a plurality of training farmland regions of interest; arranging the plurality of training farmland regions of interest according to sample dimensions into a training farmland region feature tensor; performing global eigenvalue mean pooling on the farmland regions of interest at each position in the training farmland region feature tensor to obtain the training panoramic farmland channel feature vector; The key frames in the farmland infrared video are extracted to obtain a plurality of training infrared key frames; the plurality of training infrared key frames are passed through a spatial attention feature extractor based on a convolutional neural network to obtain the training infrared image feature vector; a training farmland classification feature vector between the training panoramic farmland channel feature vector and the training infrared image feature vector is constructed; a geometric non-rigid unification factor based on orderliness prior between the training panoramic farmland channel feature vector and the training infrared image feature vector is calculated; the training farmland classification feature vector is passed through a classifier to obtain a classification loss function value; and the weighted sum between the classification loss function value and the geometric non-rigid unification factor based on orderliness prior is used as the loss function value to train the multi-layer convolutional layers, the convolutional neural network model as a target detection network, the spatial attention mechanism using a convolutional neural network, and the classifier.

[0054] In a specific embodiment of the present application, calculating the geometric non-rigid consistency factor based on orderliness prior between the training panoramic farmland channel feature vector and the training infrared image feature vector includes: calculating the geometric non-rigid consistency factor based on orderliness prior between the training panoramic farmland channel feature vector and the training infrared image feature vector using the following geometric non-rigid consistency factor formula; wherein the geometric non-rigid consistency factor formula is:

[0055]

[0056] Among them, V1 represents the training panoramic farmland channel feature vector, V2 represents the training infrared image feature vector, ||·||2 represents the bi-norm of the feature vector, cos(V1, V2) represents the calculation of the cosine distance between the training panoramic farmland channel feature vector and the training infrared image feature vector, and Loss represents the geometric non-rigid unification factor based on order prior.

[0057] In particular, in the technical solution of the present application, in order to better integrate and utilize two different types of image information, it is necessary to further improve the responsiveness between the training panoramic farmland channel feature vector and the training infrared image feature vector, considering that the training panoramic farmland channel feature vector captures the farmland features in the visible light image, such as vegetation distribution, soil conditions, etc. The training infrared image feature vector reflects the thermal distribution of the farmland and the physiological state of the vegetation. The two types of information are complementary. By improving the responsiveness between them, the two types of information can be better integrated, so that the feature vector can describe the state of the farmland more comprehensively and accurately. At the same time, the training panoramic farmland channel feature vector and the training infrared image feature vector represent different image modalities. By improving the responsiveness between them, the fusion of multimodal features can be achieved. Fusion of features of different modalities can provide richer information and enhance the expressiveness of features, thereby better distinguishing different farmland areas and detecting crop pests. In addition, improving the responsiveness between the training panoramic farmland channel feature vector and the training infrared image feature vector helps to capture the correlation between the two features. The visible light features and infrared features of farmland may be correlated in some cases, for example, a certain pest may cause abnormal heat distribution of vegetation. By enhancing the responsiveness, this correlation can be better captured and the prediction and detection capabilities of crop pests can be improved. The training panoramic farmland channel feature vector and the training infrared image feature vector constitute the training farmland classification feature vector, which is used for the classification of crop pest warning. By improving the responsiveness between them, the classifier's ability to distinguish feature vectors can be improved, thereby improving the accuracy and reliability of classification.

[0058] That is, improving the responsiveness between the training panoramic farmland channel feature vectors and the training infrared image feature vectors can make full use of the information of different image modalities, fuse multimodal features, capture the correlation between features, and improve the classifier performance, thereby improving the accuracy and reliability of crop pest prediction and detection.

[0059] Furthermore, in order to improve the viewpoint relevance of the perceptible prediction results of the feature distribution under the class probability, the geometric non-rigid unification factor based on the order prior is used as the loss function value to measure the similarity and difference between the training panoramic farmland channel feature vector and the training infrared image feature vector. Specifically, the geometric non-rigid unification factor based on the order prior is used to reduce the difficulty of mutual representation between vectors with each other's feature distribution, so as to enhance the expression ability and distinction ability of the feature vector. At the same time, the geometric non-rigid unification factor based on the order prior can also take into account the order of the feature vector, that is, the arrangement and transformation of the feature vector should maintain a certain order and regularity to conform to the inherent structure and logic of the data. Accordingly, by introducing this factor, the parameter update of the model can optimize the correlation response degree between the training panoramic farmland channel feature vector and the training infrared image feature vector, thereby improving the generalization ability and robustness of the model.

[0060] In summary, the embodiment of the present application first obtains a panoramic image of a farmland captured by a high-resolution camera and an infrared video of a farmland captured by an infrared camera, and then obtains a panoramic farmland channel feature vector after passing the panoramic image of the farmland through multiple convolutional layers by using a convolutional neural network model of a target detection network and performing arrangements and global eigenvalue mean pooling operations. Next, the key frames of the farmland infrared video are extracted and passed through a spatial attention feature extractor based on a convolutional neural network to obtain an infrared image feature vector. Finally, the panoramic farmland channel feature vector and the infrared image feature vector are fused and a classifier is used to determine whether to issue a crop pest warning prompt, so as to improve the effect of farmland pest monitoring, thereby providing more accurate warning prompts for crop protection and management, and thus helping to take timely measures to avoid losses.

[0061] Exemplary Systems

[0062] Figure 2 FIG. 1 is a block diagram of an image target detection system based on a drone according to an embodiment of the present application. Figure 2 As shown, according to the embodiment of the present application, the image target detection system 100 based on the drone includes: a farmland visual data acquisition module 110, which is used to obtain a panoramic image of the farmland acquired by a high-resolution camera and an infrared video of the farmland acquired by an infrared camera; a farmland visual feature extraction module 120, which is used to extract a panoramic farmland channel feature vector and an infrared image feature vector from the farmland panoramic image and the farmland infrared video; a construction module 130, which is used to construct a farmland classification feature vector between the panoramic farmland channel feature vector and the infrared image feature vector; and a pest result generation module 140, which is used to determine whether to issue a crop pest warning prompt based on the farmland classification feature vector.

[0063] Figure 3 FIG. 1 is a schematic diagram of the architecture of an image target detection system based on a drone according to an embodiment of the present application. Figure 3 As shown, according to the embodiment of the present application, the image target detection system 100 based on the drone includes: first, obtaining a panoramic image of a farmland captured by a high-resolution camera and an infrared video of a farmland captured by an infrared camera, then, passing the panoramic image of the farmland through multiple convolution layers to obtain a panoramic image feature map, then, passing the panoramic image feature map through a convolutional neural network model using a target detection network to obtain multiple farmland regions of interest, then, arranging the multiple farmland regions of interest according to the sample dimension into a farmland region feature tensor, and then, for each position of the farmland region feature tensor, The global eigenvalue mean pooling is performed on the region to obtain the panoramic farmland channel feature vector. Furthermore, the key frames in the farmland infrared video are extracted to obtain a plurality of infrared key frames. Furthermore, the plurality of infrared key frames are passed through a spatial attention feature extractor based on a convolutional neural network to obtain the infrared image feature vector. In particular, a farmland classification feature vector is constructed between the panoramic farmland channel feature vector and the infrared image feature vector. Finally, the farmland classification feature vector is passed through a classifier to obtain a classification result, and the classification result is used to determine whether to issue a crop pest warning prompt.

[0064] As described above, the drone-based image target detection system 100 according to the embodiment of the present application can be implemented in various terminal devices, such as a server deployed with a drone-based image target detection control algorithm. In one example, the drone-based image target detection system 100 can be integrated into the terminal device as a software module and / or a hardware module. For example, the drone-based image target detection system 100 can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the drone-based image target detection system 100 can also be one of the many hardware modules of the terminal device.

[0065] Alternatively, in another example, the drone-based image target detection system 100 and the terminal device may also be separate devices, and the drone-based image target detection system 100 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0066] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned drone-based image target detection system have been referred to above. Figure 1 The description of the UAV-based image target detection method has been introduced in detail, and therefore, its repeated description will be omitted.

[0067] Exemplary Electronic Devices

[0068] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned drone-based image target detection method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0069] See also Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0070] The processor 11 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in the embodiment of the present application; the memory 12 can be implemented in the form of a read-only memory (Read Only Memory, ROM), a static storage device, a dynamic storage device or a random access memory (Random Access Memory, RAM), etc.

[0071] The memory 12 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 12, and the processor 11 is called to execute the image target detection method based on a drone in the embodiment of this application;

[0072] Input / output interface 13, used to implement information input and output;

[0073] The communication interface 14 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0074] A bus 15 that transmits information between the various components of the device (e.g., the processor 11, the memory 12, the input / output interface 13, and the communication interface 14);

[0075] The processor 11 , the memory 12 , the input / output interface 13 and the communication interface 14 are connected to each other in communication within the device via a bus 15 .

[0076] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned drone-based image target detection method is implemented.

[0077] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0078] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

Claims

1. A method for detecting an image target based on a drone, characterized in that: include: Obtain panoramic images of farmland collected by high-resolution cameras and infrared videos of farmland obtained by infrared cameras; Extracting a panoramic farmland channel feature vector and an infrared image feature vector from the farmland panoramic image and the farmland infrared video; Constructing a farmland classification feature vector between the panoramic farmland channel feature vector and the infrared image feature vector; Based on the farmland classification feature vector, determining whether to issue a crop pest warning prompt; It also includes training methods for training multiple layers of convolutional layers, convolutional neural network models using object detection networks, spatial attention mechanisms based on convolutional neural networks, and classifiers; Among them, the training method for training the multi-layer convolutional layer, the convolutional neural network model using the target detection network, the spatial attention mechanism based on the convolutional neural network and the classifier includes: Acquire training data, wherein the training data includes a panoramic image of a training farmland acquired by a high-resolution camera and an infrared video of the training farmland acquired by an infrared camera; Passing the training farmland panoramic image through multiple convolutional layers to obtain a training panoramic image feature map; The training panoramic image feature map is used through a convolutional neural network model of a target detection network to obtain a plurality of training farmland regions of interest; Arranging the plurality of training farmland regions of interest according to sample dimensions into a training farmland region feature tensor; Performing global feature value mean pooling on the farmland region of interest at each position in the training farmland region feature tensor to obtain a training panoramic farmland channel feature vector; Extracting key frames from the training farmland infrared video to obtain a plurality of training infrared key frames; Passing the plurality of training infrared key frames through a spatial attention feature extractor based on a convolutional neural network to obtain a training infrared image feature vector; Constructing a training farmland classification feature vector between the training panoramic farmland channel feature vector and the training infrared image feature vector; Calculating a geometric non-rigid unification factor based on orderliness prior between the training panoramic farmland channel feature vector and the training infrared image feature vector; Passing the training farmland classification feature vector through a classifier to obtain a classification loss function value; Taking the weighted sum of the classification loss function value and the geometric non-rigid uniformization factor based on orderliness prior as the loss function value, training the multi-layer convolutional layer, the convolutional neural network model using the target detection network, the spatial attention mechanism based on the convolutional neural network, and the classifier; The step of calculating the geometric non-rigid unification factor between the training panoramic farmland channel feature vector and the training infrared image feature vector based on orderliness prior includes: The geometric non-rigid uniformity factor based on order prior between the training panoramic farmland channel feature vector and the training infrared image feature vector is calculated by the following geometric non-rigid uniformity factor formula; The geometric non-rigid uniformization factor formula is: ； in, represents the training panoramic farmland channel feature vector, represents the training infrared image feature vector, represents the bi-norm of the eigenvector, represents calculating the cosine distance between the training panoramic farmland channel feature vector and the training infrared image feature vector, represents the geometric non-rigid uniformization factor based on the order prior.

2. The method for detecting target in an image based on a drone according to claim 1, characterized in that: Performing feature extraction on the farmland panoramic image and the farmland infrared video to obtain a panoramic farmland channel feature vector and an infrared image feature vector, including: Passing the farmland panoramic image through multiple convolutional layers to obtain a panoramic image feature map; Performing feature extraction on the panoramic image feature map to obtain the panoramic farmland channel feature vector; Feature extraction is performed on the farmland infrared video to obtain the infrared image feature vector.

3. The method for detecting target using drone image according to claim 2, characterized in that: Extracting features from the panoramic image feature map to obtain the panoramic farmland channel feature vector includes: The panoramic image feature map is used to obtain a plurality of farmland regions of interest by using a convolutional neural network model of a target detection network; Dimensionality reduction is performed on the multiple farmland regions of interest to obtain the panoramic farmland channel feature vector.

4. The method for detecting target using drone image according to claim 3, characterized in that: Performing dimensionality reduction on the multiple farmland regions of interest to obtain the panoramic farmland channel feature vector includes: Arranging the plurality of farmland regions of interest according to sample dimensions into a farmland region feature tensor; Global eigenvalue mean pooling is performed on the farmland region of interest at each position in the farmland region feature tensor to obtain the panoramic farmland channel feature vector.

5. The method for detecting target in an image based on a drone according to claim 4, characterized in that: Extracting features from the farmland infrared video to obtain the infrared image feature vector includes: Extracting key frames from the farmland infrared video to obtain a plurality of infrared key frames; The multiple infrared key frames are passed through a spatial attention feature extractor based on a convolutional neural network to obtain the infrared image feature vector.

6. A drone-based image target detection system according to the drone-based image target detection method according to claim 1, characterized in that: include: The farmland visual data acquisition module is used to obtain the farmland panoramic images acquired by the high-resolution camera and the farmland infrared video acquired by the infrared camera; A farmland visual feature extraction module is used to extract a panoramic farmland channel feature vector and an infrared image feature vector from the farmland panoramic image and the farmland infrared video; A construction module, used to construct a farmland classification feature vector between the panoramic farmland channel feature vector and the infrared image feature vector; The pest result generation module is used to determine whether to issue a crop pest warning prompt based on the farmland classification feature vector.

7. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 5 when executing the computer program.

Images