Walnut fruit image processing method, system and storage medium based on improved YOLOv5 model

Abstract

The present invention relates to the field of computer vision technology, and in particular to a walnut fruit image processing method, system and storage medium based on an improved YOLOv5 model. The method comprises: obtaining unmanned aerial vehicle (UAV) collected walnut fruit imaging data to be processed in a walnut producing area; inputting the walnut fruit imaging data to be processed into an improved YOLOv5 model, wherein the backbone network part of the improved YOLOv5 model includes a multi-head self-attention mechanism for cross-space learning, and the feature extraction part includes a feature fusion module of a sparsely connected progressive feature fusion pyramid structure; obtaining the target walnut fruit imaging data output after being processed by the improved YOLOv5 model, so as to analyze the walnut fruit situation in the walnut producing area based on the target walnut fruit imaging data. The invention aims to solve the problem of how to reduce the computing power of an airborne edge computing platform when processing walnut fruit images.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a walnut fruit image processing method, system and storage medium based on an improved YOLOv5 model. Background Art

[0002] In the walnut production process, the task of measuring the yield of walnut trees can provide important data support for agricultural planning. Agricultural managers can optimize the allocation of agricultural resources, arrange the planting structure reasonably, and improve the efficiency of agricultural production based on the results of the yield measurement. Farmers can also apply fertilizers and irrigation more accurately by understanding the walnut yield. This helps to avoid wasting agricultural resources, improve land use efficiency, and reduce negative impacts on the environment.

[0003] In recent years, unmanned aerial vehicles (UAVs) can carry more and more powerful imaging equipment, providing more and more high-quality aerial image data for UAV aerial target detection. Currently, UAVs are widely used in military, rescue, agriculture, monitoring and other fields. Due to their higher flight capabilities and flexibility, they can collect aerial image information with richer perspectives, heights and positions. At the same time, as one of the basic problems in computer vision, the field of target detection has gradually undergone a paradigm shift in traditional machine learning methods and hand-designed features with the emergence of deep learning, and has experienced a transition from the traditional method stage to the deep learning method stage. The traditional method requires manual feature design, which is cumbersome and has poor results. Now deep learning-based methods have taken a dominant position, and a large number of new deep learning methods are being developed, which also shows the future potential of this field. With the rapid development of UAV and deep learning, the combination of the two can produce a more efficient way of working. Due to the poor visual features and more noise of small targets, small target detection has become one of the most challenging tasks in computer vision.

[0004] Currently, existing deep learning algorithms for small targets all use relatively complex network structures to extract and fuse features before inference. The models are highly complex and require huge amounts of computation, and cannot meet real-time requirements. They cannot be deployed on resource-poor edge devices such as drones for detection tasks.

[0005] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of the present invention is to provide a walnut fruit image processing method based on an improved YOLOv5 model, aiming to solve the problem of how to reduce the computing power of an airborne edge computing platform when processing walnut fruit images.

[0007] To achieve the above object, the present invention provides a walnut fruit image processing method based on an improved YOLOv5 model, the method comprising:

[0008] Obtaining drone-collected imaging data of walnut fruits to be processed in walnut producing areas;

[0009] Inputting the walnut fruit imaging data to be processed into an improved YOLOv5 model, wherein the backbone network part of the improved YOLOv5 model includes a multi-head self-attention mechanism for cross-space learning, and the feature extraction part includes a feature fusion module of a sparsely connected progressive feature fusion pyramid structure;

[0010] The target walnut fruit imaging data output after being processed by the improved YOLOv5 model is obtained, so as to analyze the walnut fruit situation in the walnut producing area based on the target walnut fruit imaging data.

[0011] Optionally, before the step of acquiring the imaging data of the walnut fruits to be processed in the walnut producing area collected by the sensor carried by the drone, the method further comprises:

[0012] The environmental data of walnut producing areas and walnut fruit imaging data are collected using various types of sensors carried by drones.

[0013] Annotating and preprocessing target type data in the walnut fruit imaging data, and performing data enhancement on the processed data to obtain a complete training data set;

[0014] The architecture of the YOLOv5 model is optimized for small target detection and an attention mechanism is added. The model is trained using the training data set to obtain the improved YOLOv5 model.

[0015] Optionally, the environmental information includes surface temperature information data of the walnut producing area, terrain information data of the walnut producing area and the surrounding area, location information data of the drone, flight control information and path planning information data, the walnut fruit imaging information includes RGB images, RGB video data, visible spectrum / infrared spectrum images and visible spectrum / infrared spectrum video data, and the sensor includes:

[0016] RGB camera, used to collect RGB images and RGB video data of walnut fruits;

[0017] Infrared thermal imaging camera, used to collect surface temperature information data in walnut producing areas;

[0018] A multispectral camera for collecting visible spectrum / infrared spectrum images and visible spectrum / infrared spectrum video data of walnut fruits;

[0019] LiDAR, used to collect terrain information data of walnut producing areas and surrounding areas;

[0020] Communication equipment, used to collect the drone's location information data, flight control information and path planning information data.

[0021] Optionally, the target type data includes RGB video data and visible spectrum / infrared spectrum video data, and the steps of labeling and preprocessing the target type data in the walnut fruit imaging data, and performing data enhancement on the processed data to obtain a complete training data set include:

[0022] The acquired RGB video data and the visible spectrum / infrared spectrum video data are intercepted into images in JPG format according to a preset sampling interval, and the images are classified and stored according to the image types, and are stored in an RGB database and a multi-spectral database respectively;

[0023] Annotate all images;

[0024] All the annotated images are enhanced according to preset image enhancement parameters to enhance the contrast, brightness and sharpness of the images to obtain the training data set.

[0025] Optionally, the multi-head self-attention mechanism for cross-space learning includes a self-attention module;

[0026] The self-attention module transforms the input features through a normalized exponential function to obtain five matrices: query, key, value, height, and width.

[0027] Optionally, the feature fusion module of the sparsely connected progressive feature fusion pyramid structure specifically includes:

[0028] The weighted feature fusion module uses a fast normalization method and is defined by the following formula:

[0029]

[0030] in, are learnable weights that can be scalars (per feature), vectors (per channel), or multidimensional tensors (per pixel), represents the extracted features, Represents the weights of all feature maps involved in fusion, represents the sum of all weights, by Then apply the ReLU activation function to ensure ≥0, and set the learning rate = 0.0001.

[0031] Optionally, the multi-head self-attention mechanism for cross-space learning includes a self-attention module in the feature fusion process, setting is the feature vector from the nth layer to the lth layer at position (i, j), and the final feature vector is the eigenvector , and The linear combination of is expressed as:

[0032]

[0033] in, , and Represents the spatial weights of the three hierarchical elements on the lth layer, satisfying the constraints:

[0034] .

[0035] In addition, to achieve the above-mentioned purpose, the present invention also provides a walnut yield information management system. The walnut yield information management system includes: an onboard edge computing platform, a memory, a processor, and a walnut fruit image processing program based on an improved YOLOv5 model stored on the memory and runnable on the processor. When the walnut fruit image processing program based on the improved YOLOv5 model is executed by the processor, the steps of the walnut fruit image processing method based on the improved YOLOv5 model as described above are implemented.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which is stored a walnut fruit image processing program based on an improved YOLOv5 model. When the walnut fruit image processing program based on the improved YOLOv5 model is executed by a processor, the steps of the walnut fruit image processing method based on the improved YOLOv5 model as described above are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the architecture of the hardware operating environment of the walnut yield information management system involved in an embodiment of the present invention;

[0038] Figure 2 It is a flow chart of a first embodiment of a walnut fruit image processing method based on an improved YOLOv5 model of the present invention;

[0039] Figure 3 A schematic diagram of a multi-head self-attention module for cross-space learning involved in an embodiment of the present invention;

[0040] Figure 4A schematic diagram of a feature fusion module of a sparsely connected progressive feature fusion pyramid structure according to an embodiment of the present invention;

[0041] Figure 5 This is a schematic diagram of a process involved in the second embodiment of the present invention;

[0042] Figure 6 This is a schematic diagram of the architecture of a walnut yield information management system according to a third embodiment of the present invention.

[0043] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0044] In order to better understand the above technical solution, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0045] As an implementation plan, Figure 1 The present invention is a schematic diagram of the hardware operating environment of the walnut yield information management system involved in the embodiment of the present invention.

[0046] like Figure 1 As shown, the walnut yield information management system may include: a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0047] Those skilled in the art will understand that Figure 1 The walnut yield information management system architecture shown in does not constitute a limitation on the walnut yield information management system, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0048] like Figure 1As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a walnut fruit image processing program based on an improved YOLOv5 model. Among them, the operating system is a program for managing and controlling the hardware and software resources of the walnut yield information management system, the walnut fruit image processing program based on the improved YOLOv5 model, and the operation of other software or programs.

[0049] exist Figure 1 In the walnut yield information management system shown, the user interface 1003 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 1004 is mainly used for the background server and communicates data with the background server; the processor 1001 can be used to call the walnut fruit image processing program based on the improved YOLOv5 model stored in the memory 1005.

[0050] In this embodiment, the walnut yield information management system includes: a memory 1005, a processor 1001, and a walnut fruit image processing program based on an improved YOLOv5 model stored in the memory and executable on the processor, wherein:

[0051] When the processor 1001 calls the walnut fruit image processing program based on the improved YOLOv5 model stored in the memory 1005, the following operations are performed:

[0052] Obtaining drone-collected imaging data of walnut fruits to be processed in walnut producing areas;

[0053] Inputting the walnut fruit imaging data to be processed into an improved YOLOv5 model, wherein the backbone network part of the improved YOLOv5 model includes a multi-head self-attention mechanism for cross-space learning, and the feature extraction part includes a feature fusion module of a sparsely connected progressive feature fusion pyramid structure;

[0054] The target walnut fruit imaging data output after being processed by the improved YOLOv5 model is obtained, so as to analyze the walnut fruit situation in the walnut producing area based on the target walnut fruit imaging data.

[0055] When the processor 1001 calls the walnut fruit image processing program based on the improved YOLOv5 model stored in the memory 1005, the following operations are performed:

[0056] The environmental data of walnut producing areas and walnut fruit imaging data are collected using various types of sensors carried by drones.

[0057] Annotating and preprocessing target type data in the walnut fruit imaging data, and performing data enhancement on the processed data to obtain a complete training data set;

[0058] The architecture of the YOLOv5 model is optimized for small target detection and an attention mechanism is added. The model is trained using the training data set to obtain the improved YOLOv5 model.

[0059] When the processor 1001 calls the walnut fruit image processing program based on the improved YOLOv5 model stored in the memory 1005, the following operations are performed:

[0060] The acquired RGB video data and the visible spectrum / infrared spectrum video data are intercepted into images in JPG format according to a preset sampling interval, and the images are classified and stored according to the image types, and are stored in an RGB database and a multi-spectral database respectively;

[0061] Annotate all images;

[0062] All the annotated images are enhanced according to preset image enhancement parameters to enhance the contrast, brightness and sharpness of the images to obtain the training data set.

[0063] Based on the hardware architecture of the walnut yield information management system using the above-mentioned computer vision technology, an embodiment of the walnut fruit image processing method based on the improved YOLOv5 model of the present invention is proposed.

[0064] First embodiment

[0065] Reference Figure 2 In the first embodiment, the walnut fruit image processing method based on the improved YOLOv5 model includes the following steps:

[0066] Step S10, obtaining imaging data of walnut fruits to be processed in a walnut producing area collected by a drone;

[0067] In this embodiment, imaging data of walnut fruits to be processed in a walnut producing area collected by a drone is obtained.

[0068] Optionally, in order to enable the drone to collect high-precision walnut fruit imaging data, the deployment operation is performed on the website configured for the walnut yield information management system, and it is deployed to the cloud server in a front-end and back-end separated manner. The system environment Ubuntu is installed on the edge computing device, and the Pytorch model inference environment is configured. The drone API interface is configured, and the connection between the drone and the edge computing device is established, and it is installed on the drone body; then the drone route flight mission is planned to ensure that the camera can obtain real-time video stream data, configure the drone API interface, obtain the drone real-time video data stream (h264), and transcode the video data stream to MP4 format, deploy the model to run, perform detection operations on the transcoded real-time video stream, and the edge computing device transmits various detection results and detection videos back to the website for display.

[0069] Optionally, the drone is provided with various types of sensors to collect environmental data of the walnut producing area and walnut fruit imaging data. RGB camera, infrared thermal imaging camera, multispectral camera, laser radar and communication equipment, wherein the RGB camera is used to collect RGB images and RGB video data of walnut fruits; the infrared thermal imaging camera is used to collect surface temperature information data of the walnut producing area; the multispectral camera is used to collect visible spectrum / infrared spectrum images and visible spectrum / infrared spectrum video data of walnut fruits; the laser radar is used to collect terrain information data of the walnut producing area and the surrounding area; the communication equipment is used to collect the location information data, flight control information and path planning information data of the drone.

[0070] Among them, RGB images, RGB video data, visible spectrum / infrared spectrum images, and visible spectrum / infrared spectrum video data are used to facilitate the walnut production information management system to identify walnut fruit characteristics from image data; surface temperature information data is used for the walnut production information management system to determine the surface temperature in the production area, so as to analyze the planting environment in the walnut production area. Terrain information data, location information data, flight control information and path planning information data are used for drones to perform flight missions.

[0071] Step S20, inputting the walnut fruit imaging data to be processed into an improved YOLOv5 model, wherein the backbone network part of the improved YOLOv5 model includes a multi-head self-attention mechanism for cross-space learning, and the feature extraction part includes a feature fusion module of a sparsely connected progressive feature fusion pyramid structure;

[0072] In this embodiment, the acquired walnut fruit imaging data to be processed is input into the improved YOLOv5 model, wherein the backbone network part of the improved YOLOv5 model includes a multi-head self-attention mechanism for cross-space learning, and the feature extraction part includes a feature fusion module of a sparsely connected progressive feature fusion pyramid structure.

[0073] Optionally, the multi-head self-attention mechanism for cross-space learning includes a self-attention module, which transforms the input features through a normalized exponential function to obtain five matrices of query, key, value, height and width.

[0074] Specifically, refer to Figure 3 Schematic diagram of the Cross Spatial Learning-Multi Head Self Attention (CSL-MHSA) module shown:

[0075] CSL-MHSA assigns weights to various features based on their relevance, directing the network's attention to the target of interest, thereby reducing interference from irrelevant backgrounds and enhancing the network's feature extraction performance. The multi-head self-attention mechanism consists of multiple self-attention layers in parallel, where and, i = 1, 2, ..., represent the input and output feature spaces of the self-attention layer, respectively. The input of the self-attention module is transformed by linear operations to obtain five matrices: query (W Q ), key (W K ), value (W V )、Height(R h ) and width (R w ), calculate W Q and W K , denoted as W Q·K , the calculation formula of this module is as follows:

[0076]

[0077] Among them, the Softmax function is a normalized exponential function that represents the similarity of query, key, height, and width in the form of probability. To ensure the non-negativity of the value, the similarity value is transformed using an exponential function. To ensure that the sum of all similarity values ​​is 1, the output result represents the proportion of its exponential value in the sum, as shown in the equation:

[0078]

[0079] Where f is a vector, f y and f c is one of the elements.

[0080] Optionally, refer to Figure 4 The schematic diagram of the feature fusion module of the sparsely connected progressive feature fusion pyramid structure is shown: The feature fusion module of the sparsely connected progressive feature fusion pyramid structure specifically includes: a weighted feature fusion module. As the main network gradually extracts features layer by layer, the sparsely connected progressive feature fusion pyramid structure incrementally fuses the extracted feature maps at low, medium and high levels. The process includes initially fusing low-level features, and then gradually fusing medium-level features and high-level features using the Mild intermediate layer. It should be noted that during the fusion process, the weighted feature fusion module uses a fast normalization method, defined by the following formula:

[0081]

[0082] in, are learnable weights that can be scalars (per feature), vectors (per channel), or multidimensional tensors (per pixel), represents the extracted features, Represents the weights of all feature maps involved in fusion, represents the sum of all weights, by Then apply the ReLU activation function to ensure ≥0, and set the learning rate = 0.0001.

[0083] Optionally, the feature fusion process adopts an adaptive spatial fusion method, and the multi-head self-attention mechanism of cross-space learning includes a self-attention module in the feature fusion process. is the feature vector from the nth layer to the lth layer at position (i, j), and the final feature vector is the eigenvector , and The linear combination of is expressed as:

[0084]

[0085] in, , and Represents the spatial weights of the three hierarchical elements on the lth layer, satisfying the constraints:

[0086]

[0087] Step S30, obtaining the target walnut fruit imaging data output after being processed by the improved YOLOv5 model, so as to analyze the walnut fruit situation in the walnut producing area based on the target walnut fruit imaging data.

[0088] In this embodiment, the walnut fruit imaging data to be processed is processed by the improved YOLOv5 model to obtain the target walnut fruit imaging data.

[0089] It should be noted that after the target walnut fruit imaging data is processed by the YOLOv5 model, the walnut yield information management system can better capture the feature relationship between different regions in the image during the subsequent image analysis due to the multi-head self-attention mechanism after cross-space learning, thereby improving the model's detection accuracy for walnut fruits in complex backgrounds and reducing false detections and missed detections; in addition, cross-space learning enhances the YOLOv5 model's ability to learn features at different spatial positions, enabling the model to better understand and parse the image information of walnut fruits.

[0090] For the feature fusion module of the sparsely connected progressive feature fusion pyramid structure in the feature extraction part of the improved YOLOv5 model, due to the sparsely connected progressive feature fusion pyramid structure, the model can better capture the multi-scale features of walnut fruits and improve the detection ability of walnut fruits of different sizes and shapes; and by progressively fusing features at different levels, the model can extract richer and more detailed image features from the target walnut fruit imaging data, which helps to more accurately identify the details of walnut fruits. Sparse connections reduce the computational complexity of the model, so that the model can more quickly process a large amount of walnut fruit image data collected by drones while ensuring detection accuracy.

[0091] In the technical solution provided in this embodiment,

[0092] Second embodiment

[0093] As an optional embodiment, in this embodiment, in order to enable the improved YOLOv5 model to accurately process the walnut fruit imaging data to be processed, the target type data in the walnut fruit imaging data is labeled and preprocessed, the processed data is enhanced to obtain a complete training data set, the architecture is optimized in the direction of small target detection on the YOLOv5 model architecture, and an attention mechanism is added, and the model is trained using the training data set to obtain the improved YOLOv5 model.

[0094] Further and optionally, reference is made to Figure 5 The steps of labeling and preprocessing the target type data in the walnut fruit imaging data and performing data enhancement on the processed data to obtain a complete training data set specifically include:

[0095] Step S100, the acquired RGB video data and the visible spectrum / infrared spectrum video data are intercepted into images in JPG format according to a preset sampling interval, and the images are classified and stored according to the image types, and stored in an RGB database and a multi-spectral database respectively;

[0096] Step S200, labeling all images;

[0097] Step S300: For all the annotated images, the preset image enhancement parameters are used to enhance the contrast, brightness and sharpness of the images to obtain the training data set.

[0098] Third embodiment

[0099] As an optional embodiment, refer to Figure 6 In this embodiment, a walnut yield information management system is also provided, and the system includes the following modules:

[0100] The data collection module 100 is used to collect relevant data of the walnut producing area in real time, including the growth environment, soil conditions, climate data, etc.; and collect information such as the growth of walnut trees and the maturity of fruits through sensors and monitoring equipment;

[0101] The walnut yield prediction module 200 is used to predict the walnut yield based on historical data and current environmental factors; and to use data analysis and machine learning algorithms to achieve prediction and trend analysis of future yields;

[0102] The data analysis and reporting module 300 is used to analyze and compile statistics on the collected data, generate production reports and analysis reports, and provide data visualization functions to help users understand the production situation and production trends;

[0103] The real-time monitoring and remote control module 400 is used to realize the real-time monitoring and remote control of the walnut producing area, including remote monitoring of equipment status, adjustment of production parameters, etc.

[0104] The data security and backup module 500 is used to ensure the security and integrity of the walnut production information, take data encryption and backup measures to prevent data loss and leakage, and ensure the confidentiality and reliability of the walnut production information.

[0105] In addition, it can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing related hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the walnut yield information management system to implement the process steps of the embodiment of the above method.

[0106] Therefore, the present invention also provides a computer-readable storage medium, which stores a walnut fruit image processing program based on an improved YOLOv5 model. When the walnut fruit image processing program based on the improved YOLOv5 model is executed by a processor, the various steps of the walnut fruit image processing method based on the improved YOLOv5 model described in the above embodiment are implemented.

[0107] The computer-readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0108] It should be noted that since the storage medium provided in the embodiment of the present application is the storage medium used to implement the method of the embodiment of the present application, based on the method introduced in the embodiment of the present application, the person skilled in the art can understand the specific structure and deformation of the storage medium, so it is not repeated here. All storage media used in the method of the embodiment of the present application belong to the scope of protection of this application.

[0109] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0113] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0114] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0115] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A walnut fruit image processing method based on an improved YOLOv5 model, It is characterized in that Applied to an airborne edge computing platform, the method comprises the following steps: Obtaining drone-collected imaging data of walnut fruits to be processed in walnut producing areas; Inputting the walnut fruit imaging data to be processed into an improved YOLOv5 model, wherein the backbone network part of the improved YOLOv5 model includes a multi-head self-attention mechanism for cross-space learning, and the feature extraction part includes a feature fusion module of a sparsely connected progressive feature fusion pyramid structure; Obtaining the target walnut fruit imaging data output after being processed by the improved YOLOv5 model, so as to analyze the walnut fruit situation in the walnut producing area based on the target walnut fruit imaging data; The multi-head self-attention mechanism for cross-space learning includes a self-attention module; The self-attention module transforms the input features through a normalized exponential function to obtain five matrices: query, key, value, height, and width; The feature fusion module of the sparsely connected progressive feature fusion pyramid structure specifically includes: The weighted feature fusion module uses a fast normalization method and is defined by the following formula: ； in, are learnable weights, represents the extracted features, Represents the weights of all feature maps involved in fusion, represents the sum of all weights, by Then apply the ReLU activation function to ensure ≥0, and set the learning rate = 0.0001.

2. The method according to claim 1, It is characterized in that Before the step of acquiring the imaging data of the walnut fruits to be processed in the walnut producing area collected by the sensor carried by the drone, the method further includes: The environmental data of walnut producing areas and walnut fruit imaging data are collected using various types of sensors carried by drones. Annotating and preprocessing target type data in the walnut fruit imaging data, and performing data enhancement on the processed data to obtain a complete training data set; The architecture of the YOLOv5 model is optimized for small target detection and an attention mechanism is added. The model is trained using the training data set to obtain the improved YOLOv5 model.

3. The method according to claim 2, It is characterized in that The environmental data includes surface temperature information data of the walnut producing area, terrain information data of the walnut producing area and the surrounding area, location information data of the drone, flight control information and path planning information data, the walnut fruit imaging data includes RGB images, RGB video data, visible spectrum / infrared spectrum images and visible spectrum / infrared spectrum video data, and the sensors include: RGB camera, used to collect RGB images and RGB video data of walnut fruits; Infrared thermal imaging camera, used to collect surface temperature information data in walnut producing areas; A multispectral camera for collecting visible spectrum / infrared spectrum images and visible spectrum / infrared spectrum video data of walnut fruits; LiDAR, used to collect terrain information data of walnut producing areas and surrounding areas; Communication equipment, used to collect the drone's location information data, flight control information and path planning information data.

4. The method according to claim 2, It is characterized in that The target type data includes RGB video data and visible spectrum / infrared spectrum video data, and the steps of labeling and preprocessing the target type data in the walnut fruit imaging data, and performing data enhancement on the processed data to obtain a complete training data set include: The acquired RGB video data and the visible spectrum / infrared spectrum video data are intercepted into images in JPG format according to a preset sampling interval, and the images are classified and stored according to the image types, and are stored in an RGB database and a multi-spectral database respectively; Annotate all images; All the annotated images are enhanced according to preset image enhancement parameters to enhance the contrast, brightness and sharpness of the images to obtain the training data set.

5. The method according to claim 1, It is characterized in that The multi-head self-attention mechanism of cross-space learning includes a self-attention module in the feature fusion process. is the feature vector from the nth layer to the lth layer at position (i, j), and the final feature vector is the eigenvector , and The linear combination of is expressed as: ； in, , and Represents the spatial weights of the three hierarchical elements on the lth layer, satisfying the constraints: 。 6. A walnut yield information management system, It is characterized in that The walnut yield information management system includes: an onboard edge computing platform, a memory, a processor, and a walnut fruit image processing program based on an improved YOLOv5 model stored in the memory and runnable on the processor. When the walnut fruit image processing program based on the improved YOLOv5 model is executed by the processor, the steps of the walnut fruit image processing method based on the improved YOLOv5 model as described in any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a walnut fruit image processing program based on the improved YOLOv5 model. When the walnut fruit image processing program based on the improved YOLOv5 model is executed by the processor, the steps of the walnut fruit image processing method based on the improved YOLOv5 model as described in any one of claims 1 to 5 are implemented.

Images