Method, system, equipment and medium for detecting longan fruit bunches in complex orchards based on image recognition

Abstract

The present invention discloses a method, system, device and medium for detecting longan fruit bunches in a complex orchard based on image recognition, and the method comprises: constructing an image recognition model; the image recognition model comprises a pre-training parameter loading module, an image size normalization module, a target feature extraction module, a pooling module, a multi-scale feature fusion module and a target prediction module; performing data amplification and preprocessing on the collected longan fruit bunches images to obtain an initial data set; annotating the longan fruit bunches on each image in the initial data set; dividing the annotated initial data set into a training set and a verification set according to a set ratio; training the image recognition model using the images and annotated data in the training set; and detecting the longan fruit bunches in the longan orchard image using the trained longan fruit bunches detection model. The present invention applies computer vision technology to the detection of longans in complex orchards under natural conditions, avoiding the limitations of recognition methods based on shape, color thresholds and the like.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a method, system, equipment and medium for detecting longan fruit bunches in a complex orchard based on image recognition. Background Art

[0002] Longan is a famous fruit in tropical and subtropical regions with important economic value. Due to its short fruiting period, timely picking of longan is crucial for successful harvesting. Currently, manual harvesting is the main method. This harvesting method is not only laborious and time-consuming, but also with the transfer of rural labor to non-agricultural industries, the surplus labor force in rural areas is gradually decreasing. Therefore, the study of longan fruit cluster detection is of great significance for the longan intelligent picking robot to complete the picking task, so the image detection method of longan fruit cluster has become an important detection technology.

[0003] The current traditional longan detection method mainly segments the fruit and background information based on shape and color thresholds to obtain the detection results of longan. However, this method is only suitable for laboratory environments with good light and a single background. The accuracy of longan fruit bunch detection in orchard scenes with complex backgrounds and uneven lighting is low. Therefore, research on the detection of orchard longan fruit bunches based on image recognition is of great significance to the development of complex orchard intelligent equipment. Summary of the invention

[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a method, system, equipment and medium for detecting longan fruit bunches in complex orchards based on image recognition, which can accurately detect longan fruit bunches in complex orchards, improve the intelligence level of longan picking robots, reduce labor costs, and realize rapid detection of longan fruit bunches of different varieties under different lighting and shading conditions, thereby providing information for longan picking robots.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] On the one hand, the present invention provides a method for detecting longan fruit clusters in a complex orchard based on image recognition, comprising the following steps:

[0007] Constructing an image recognition model; the image recognition model includes a pre-training parameter loading module, an image size normalization module, a target feature extraction module, a pooling module, a multi-scale feature fusion module and a target prediction module; the feature extraction module adopts a MobileNetV3 module, the MobileNetV3 module is the backbone network of the image recognition model, the MobileNetV3 module uses a standard convolution and multiple bneck structures to extract features, the bneck structure combines a deep separable convolution with a residual structure, increases the feature extraction capability of longan fruit clusters, uses a convolution block instead of a fully connected layer after the feature extraction layer, and adds a maximum pooling layer to obtain the final classification result;

[0008] Acquire images of longan fruit bunches in the mature stage in a complex longan orchard scene; perform data amplification and preprocessing on the collected longan fruit bunch images to obtain an initial data set; annotate the longan fruit bunches on each image in the initial data set to obtain an information file in XML format; divide the annotated initial data set into a training set and a validation set according to a set ratio;

[0009] Use the images and annotation data in the training set to train the image recognition model;

[0010] The trained longan fruit bunch detection model is used to detect longan fruit bunches in longan orchard images.

[0011] As a preferred technical solution, the pre-training parameter loading module imports the pre-training parameters and assigns an initial weight value to the target;

[0012] The image size normalization module adjusts the image to be detected to a uniform pixel size by filling the edges without stretching the image;

[0013] The pooling module adopts an SPP module, which uses multiple convolution kernels of different sizes to perform maximum pooling processing on the front layer features, and then connects the multiple processed results to form a new feature layer, thereby increasing the network depth while retaining the front layer features to obtain more local feature information.

[0014] As a preferred technical solution, the multi-scale feature fusion module includes a feature pyramid network and a path aggregation network. The path aggregation network first upsamples the result of feature extraction of the MobileNetV3 module and then downsamples it, thereby enhancing the information extraction capability of the feature pyramid network, extracting deeper semantic information of longan fruit clusters, and detecting more refined longan fruit cluster targets.

[0015] As a preferred technical solution, the acquisition of the image of a bunch of longan fruits in the mature stage in the complex longan orchard scene is specifically as follows:

[0016] When the longan fruit is at the ripening stage, choose the daytime on sunny and cloudy days respectively;

[0017] An RGB camera was used to collect images at a distance of 0.5-1m from the longan fruit;

[0018] Images were acquired both in direct sunlight and in backlit scenes.

[0019] As a preferred technical solution, the collected longan fruit bunch images are subjected to data amplification and preprocessing to obtain an initial data set, specifically:

[0020] Data amplification is performed on the collected longan fruit bunch images, and the image amplification methods include random translation, random cropping and brightness transformation;

[0021] The collected longan fruit bunch images are preprocessed by image normalization.

[0022] As a preferred technical solution, the image recognition model is trained using the images and annotation data in the training set, specifically including:

[0023] Use the model loading module to import pre-trained parameters and assign an initial weight value to the target;

[0024] Select a predetermined number of images and their annotation data from the training set and input them into the image size normalization module to adjust the image to be detected to a uniform pixel size. When adjusting, the edge is filled in the way without stretching the image.

[0025] The images adjusted to a uniform pixel size are input into the feature extraction module to extract the deep-level longan fruit cluster features;

[0026] The feature image after the feature extraction module is operated is subjected to pooling operation and multi-scale feature fusion operation in sequence, and finally the result is predicted;

[0027] During the model training process, after each round of training, the current image recognition model is verified using the images and annotation data in the verification set, and the weight value of the current image recognition model is adjusted according to the verification results;

[0028] The maximum number of iterations is set. When the preset maximum number of iterations is reached or the error rate on the validation set no longer decreases, the training is terminated. The image recognition model with the best validation result is selected as the longan fruit string detection model.

[0029] As a preferred technical solution, the longan fruit bunch detection model trained is used to detect the longan fruit bunches in the longan orchard image, specifically:

[0030] The longan image collected by the camera in real time is input into the longan fruit bunch detection model, and is processed in sequence by the parameter loading module, image input size normalization module, target feature extraction module, pooling module, and multi-scale feature fusion module of the trained longan fruit bunch detection model. In the prediction module, the maximum circumscribed rectangular boxes of multiple candidate longan fruit bunches are output through convolution operation and the corresponding confidence values ​​are given. The areas marked by all rectangular boxes with confidence values ​​greater than 0.75 are saved to obtain the longan fruit bunch detection results.

[0031] On the other hand, the present invention provides a complex orchard longan fruit bunch detection system based on image recognition, which is applied to the complex orchard longan fruit bunch detection method based on image recognition, and includes an image recognition model construction module, a longan fruit bunch image data set acquisition module, a detection model training module, a detection recognition module and an information display module;

[0032] The image recognition model construction module includes a pre-training parameter loading module, an image size normalization module, a target feature extraction module, a pooling module, a multi-scale feature fusion module and a target prediction module; the feature extraction module adopts a MobileNetV3 module, which is the backbone network of the image recognition model. The MobileNetV3 module uses a standard convolution and multiple bneck structures to extract features. The bneck structure combines a deep separable convolution with a residual structure, increases the feature extraction capability of longan fruit clusters, uses a convolution block instead of a fully connected layer after the feature extraction layer, and adds a maximum pooling layer to obtain the final classification result;

[0033] The longan fruit bunch image data set acquisition module is used to collect longan fruit bunch images in direct sunlight and backlight scenes under natural conditions, using a camera during daytime on sunny and cloudy days to form an initial data set of longan fruit bunches, and perform data amplification and preprocessing on the initial data set of longan fruit bunches, annotate the objects in the image, and obtain a training set and a verification set of longan fruit bunches;

[0034] The detection model training module is used to import the images and annotation data in the longan fruit string training set into the target detection network model for iterative training. After each round of iterative training of the image data in the training set, the images and annotation data in the longan fruit string verification set are imported into the model to verify the performance of the model, and the weight value of the target is adjusted. When the preset maximum number of iterations is reached or the error rate on the verification set no longer decreases, the training is terminated to obtain the optimal longan fruit string detection model;

[0035] The detection and recognition module is used to load the optimal model weight value, import the longan image collected by the camera in real time into the longan fruit cluster detection and recognition model, and obtain the detection result of the longan fruit cluster;

[0036] The information display module is used to input the detection result image of the longan fruit cluster into a display device for display.

[0037] Another aspect of the present invention provides an electronic device, the electronic device comprising:

[0038] at least one processor; and,

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the complex orchard longan fruit bunch detection method based on image recognition.

[0041] On the other hand, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the complex orchard longan fruit bunch detection method based on image recognition.

[0042] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0043] Compared with the current traditional longan detection method, the longan fruit bunch detection method and system provided by the present invention based on image recognition applies computer vision technology to the detection of longan in complex orchards under natural conditions, avoiding the limitations of recognition methods based on shape, color thresholds, etc.; the generalization ability of detecting longan fruit bunches in orchard scenes with complex backgrounds and uneven lighting is relatively strong, and the efficiency is relatively high. Only an ordinary camera and a microprocessor are needed to realize real-time fast and accurate detection of longan fruit bunches of different varieties, which is convenient to be mounted on a longan picking robot, providing a pair of eyes for the longan picking robot. Therefore, research on the detection of longan fruit bunches in complex orchards based on image recognition is of great significance to the development of intelligent orchard equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 is a flow chart of a longan fruit cluster detection method based on image recognition according to Embodiment 1 of the present invention;

[0046] Figure 2 is a schematic diagram of the image recognition structure of Embodiment 1 of the present invention;

[0047] Figure 3 is a PR curve result diagram of the detection and recognition model of Example 1 of the present invention;

[0048] Figure 4 Schematic diagram of the structure of a longan fruit cluster detection system based on image recognition according to Embodiment 2 of the present invention;

[0049] Figure 5 It is a schematic diagram of the structure of the microprocessor and related devices of Example 3 of the present invention. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0051] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0052] Embodiment 1:

[0053] like Figure 1 As shown, this embodiment provides a method for detecting longan fruit clusters in a complex orchard based on image recognition, comprising the following steps:

[0054] S101, constructing an image recognition model;

[0055] Specifically, the structure of the image recognition model is as follows: Figure 2As shown, it includes a pre-training parameter loading module, an image size normalization module, a target feature extraction module, a pooling module, a multi-scale feature fusion module and a target prediction module. The image recognition model of this embodiment adopts a convolutional neural network; the pre-training parameter loading module imports the pre-training parameters and assigns an initial weight value to the target; the image size normalization module adjusts the image to be detected to a uniform pixel size, and adopts an edge filling method during adjustment without stretching the image; the feature extraction module adopts a MobileNetV3 module, which is the backbone network of the image recognition model. The MobileNetV3 module uses a 3×3 standard convolution and multiple bneck structures to extract features. The bneck structure combines the depth separable convolution with the residual structure, which increases the feature extraction capability of the longan string fruit. After the feature extraction layer, A 1×1 convolution block is used to replace the fully connected layer, and a maximum pooling layer is added to obtain the final classification result, further reducing the amount of network parameters; the pooling module adopts the SPP module, and the SPP module uses 4 convolution kernels of different sizes to perform maximum pooling processing on the front layer features, and then connects the 4 processed results to form a new feature layer, which increases the network depth while retaining the front layer features to obtain more local feature information; the multi-scale feature fusion module includes a feature pyramid network and a path aggregation network. The path aggregation network first upsamples the result of feature extraction from the MobileNetV3 backbone network, and then downsamples it, which enhances the information extraction capability of the feature pyramid, can extract deeper longan fruit semantic information, and detect more refined longan fruit targets. When the sizes of multiple longan fruit clusters in the same picture differ greatly, accurate detection can still be achieved.

[0056] S102, acquiring an image of a bunch of longan fruits in a mature stage in a complex longan orchard scene;

[0057] Specifically, in this embodiment, when the longan fruit is in the ripening stage, sunny and cloudy daytime are selected respectively, and an RGB camera is used to collect images at a distance of 0.5-1m from the longan fruit. Images are collected in scenes under direct sunlight and backlight. The longan fruit bunch image is an RGB three-channel image, with a total of 750 images of 1280×720 pixels.

[0058] S103, performing data amplification and preprocessing on the collected longan fruit bunch images to obtain an initial data set;

[0059] Specifically, in order to increase the number of training samples to enhance the accuracy and robustness of the detector, data amplification is performed on the collected original images, including: random translation transformation (transformation range is ±20 pixels), random cropping transformation (cropping size is set to 720×720 pixels) and brightness transformation (transformation range is ±15%). After data amplification, a total of 1050 images are obtained, and the obtained images are normalized and uniformly scaled to N×N pixel images. In this embodiment, N=608, forming an initial data set.

[0060] S104, labeling the initial data set, using the LabelImg program, and framing the longan fruit bunches in the image with a rectangular labeling frame, so that the rectangular labeling frame circumscribes the longan fruit bunch area; when labeling, whether the longan fruit bunches on a certain image need to be labeled is determined by visually observing whether the longan fruit bunches are complete under conditions such as image shooting angle and lighting. If they are complete, labeling is performed; if they are incomplete, labeling is abandoned.

[0061] S105, dividing the annotated initial data set into a training set and a validation set in a ratio of 4:1;

[0062] Specifically, 840 of the 1050 annotated images are used as training sets and 210 as validation sets.

[0063] S106, training the image recognition model using the images and annotation data in the training set;

[0064] The step S106 specifically includes:

[0065] S1061, inputting the image adjusted to a uniform pixel size into a feature extraction module to extract deep-level longan fruit cluster features;

[0066] Specifically, a preset number of longan fruit images in the training set are adjusted to 416×416 pixels and input into the MobileNetV3 module. The h-swish activation function is used instead of swish in the backbone module, and the h-sigmoid activation function is used instead of sigmoid in the SE module. This can reduce the computational complexity of the network and extract deep-level longan fruit features. The size of the input image is gradually changed from 416×416 to 13×13.

[0067] S1062, inputting the feature image after the feature extraction module operation into the pooling module, and adopting the maximum pooling operation;

[0068] Specifically, the feature image after operation by the MobileNetV3 module is input into the SPP module. In the SPP module, the maximum pooling operations of 13×13, 9×9, 5×5, and 1×1 are adopted, and then the four processed results are connected to form a new feature layer. While increasing the network depth, the features of the previous layer are retained to obtain more local feature information, achieving more accuracy improvements with less computational cost.

[0069] S1063. The result after the pooling module operation is input into the path aggregation network, and first up-sampled and then down-sampled. This strengthens the information extraction capability of the feature pyramid, can extract deeper semantic information of longan fruit bunches, detect more refined longan fruit bunches targets, and can still achieve accurate detection when the sizes of multiple longan fruit bunches in the same picture differ greatly.

[0070] S1064. The image recognition model is trained iteratively until all images and annotation data in the training set are used for model training, completing one round of epoch training.

[0071] In this embodiment, the number of images in the training set is 840, the number of input images in each iteration is batch size=4, the number of iterations per epoch is equal to the number of images in the training set divided by the batch size, that is, the number of iterations per epoch=840 / 4=210, and the images in the training set need to be iterated for a total of 210 times to complete one round of epoch training.

[0072] S107, after each epoch of training, the current image recognition model is verified using the images and annotation data in the verification set, and the weight values ​​of the parameters of the current image recognition model are adjusted according to the verification results.

[0073] In this embodiment, the number of training rounds epoch=1000, and the number of iterations of each epoch training round is 210 times. The current image recognition model is verified using the images and annotation data in the verification set, and the weight values ​​of various parameters of the current image recognition model are adjusted according to the verification results, specifically including:

[0074] S1071. Use the images and annotated data in the validation set to validate the current image recognition model, set multiple different thresholds, and use the model to calculate and obtain multiple sets of different precision rates (P) and recall rates (R).

[0075] S1072. Draw a precision-recall (PR) curve based on multiple sets of different precision and recall values, with the recall as the horizontal axis and the precision as the vertical axis, and use the area enclosed by the drawn PR curve and the horizontal and vertical axes as the average precision (AP).

[0076] Specifically, the PR curve result of the detection and recognition model is as follows: Figure 3 As shown, it can be seen that the area enclosed by the PR curve and the horizontal and vertical axes almost covers the entire area, indicating that this detection model can identify longan fruit clusters from complex orchard scenes.

[0077] S1073. Adjust the weight values ​​of each parameter of the current image recognition model according to the value of the training loss function and the average accuracy of all targets.

[0078] After completing the above steps S1071 to S1073, determine whether the number of training rounds epoch reaches the preset value. If the number of training rounds epoch does not reach the preset value, that is, the number of training rounds epoch is less than 1000, return to step S106. If the number of training rounds epoch reaches the preset value, that is, the number of training rounds epoch is equal to 1000, enter step S108.

[0079] S108. Select the image recognition model with the highest average accuracy in the verification results as the longan fruit cluster detection model.

[0080] S109, using a longan fruit bunch detection model to detect longan fruit bunches in a complex orchard longan image.

[0081] In this embodiment, the longan image collected by the camera in real time is used as the complex orchard longan image. The longan fruit bunch detection model will detect the next complex orchard longan image only after detecting one complex orchard longan image. In the detection of each complex orchard longan image, each complex orchard longan image is input into the longan fruit bunch detection model, and is processed by the optimal model parameter loading module, the image input size normalization module, the target feature extraction module, the pooling module, and the multi-scale feature fusion module in sequence. The processing of the image input size normalization module, the target feature extraction module, the pooling module, and the multi-scale feature fusion module can refer to the above steps S1061 to S1064. In the prediction module, the maximum circumscribed rectangular boxes of multiple candidate longan fruit bunches are output through convolution operation and the corresponding confidence values ​​are given. The areas marked by all rectangular boxes with confidence values ​​greater than 0.75 are saved to obtain the longan fruit bunch detection results.

[0082] The method of the above embodiment adopts Ubuntu 18.04 system, Python programming language, and is implemented on a microprocessor with NVIDIA Jetson TX2 GPU, 8GB RAM and a display model of HD Graphics.

[0083] It should be noted that although the method operations of the above embodiments are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0084] Embodiment 2:

[0085] like Figure 4 As shown, this embodiment provides a complex orchard longan fruit bunch detection system based on image recognition, the system includes building an image recognition model, a longan fruit bunch image data set acquisition module, a longan fruit bunch image data amplification and preprocessing module, a detection model training module, a detection model verification module, a detection recognition module and an information display module, and the specific functions of each module are as follows:

[0086] Constructing an image recognition model, wherein the image recognition model includes a pre-training parameter loading module, an image size normalization module, a target feature extraction module, a pooling module, a multi-scale feature fusion module and a target prediction module;

[0087] The longan fruit bunch image dataset acquisition module is used to collect longan fruit bunch images in direct sunlight and backlight scenes under natural conditions, using a camera during sunny and cloudy daytime to form the longan fruit bunch initial dataset;

[0088] The longan fruit bunch image data amplification and preprocessing module is used to perform data amplification and preprocessing on the initial data set of the longan fruit bunch, mark the objects in the image, and obtain the training set and verification set of the longan fruit bunch;

[0089] A detection model training module, used for importing the images and annotation data in the longan fruit string training set into the target detection network model for iterative training;

[0090] The detection model verification module is used to import the images and annotation data in the longan fruit string verification set into the model to verify the performance of the model and adjust the weight value of the target. When the preset maximum number of iterations is reached or the error rate on the verification set no longer decreases, the training is terminated to obtain the optimal longan fruit string detection model.

[0091] A detection and recognition module is used to load the optimal model weight value, import the longan image collected by the camera in real time into the longan fruit bunch detection and recognition model, and obtain the detection result of the longan fruit bunch;

[0092] The information display module is used to input the detection result image of the longan fruit cluster into a display device for display.

[0093] It should be noted that the complex orchard longan fruit bunch detection system based on image recognition of the present invention corresponds one-to-one to the complex orchard longan fruit bunch detection method based on image recognition in Example 1 of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the complex orchard longan fruit bunch detection method based on image recognition are applicable to the embodiment of the complex orchard longan fruit bunch detection method based on image recognition. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0094] In addition, in the implementation of the complex orchard longan fruit bunch detection system based on image recognition in the above-mentioned embodiment, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the multi-party privacy protection machine learning system based on homomorphic encryption and trusted hardware is divided into different program modules to complete all or part of the functions described above.

[0095] Embodiment 3:

[0096] This embodiment provides a microcomputer device, which can be any microcomputer, such as Figure 5 As shown, the Jetson TX2 GPU processor, memory, input device, display and network interface connected by the Ubuntu18.04 system bus, the microprocessor is used to provide computing and control capabilities, the memory includes a non-volatile storage medium and an internal memory, the non-volatile storage medium stores the Ubuntu18.04 operating system, the related programs of the Python programming language and the longan image database, the internal memory provides an environment for the operation of the Ubuntu18.04 operating system and the related programs of the Python programming language in the non-volatile storage medium, when the processor executes the related programs of the Python programming language stored in the memory, the complex orchard longan fruit string detection method of the above-mentioned embodiment 1 is implemented as follows:

[0097] Constructing an image recognition model; the image recognition model includes a pre-training parameter loading module, an image size normalization module, a target feature extraction module, a pooling module, a multi-scale feature fusion module and a target prediction module; the feature extraction module adopts a MobileNetV3 module, the MobileNetV3 module is the backbone network of the image recognition model, the MobileNetV3 module uses a standard convolution and multiple bneck structures to extract features, the bneck structure combines a deep separable convolution with a residual structure, increases the feature extraction capability of longan fruit clusters, uses a convolution block instead of a fully connected layer after the feature extraction layer, and adds a maximum pooling layer to obtain the final classification result;

[0098] Acquire images of longan fruit bunches in the mature stage in a complex longan orchard scene; perform data amplification and preprocessing on the collected longan fruit bunch images to obtain an initial data set; annotate the longan fruit bunches on each image in the initial data set to obtain an information file in XML format; divide the annotated initial data set into a training set and a validation set according to a set ratio;

[0099] Use the images and annotation data in the training set to train the image recognition model;

[0100] The trained longan fruit bunch detection model is used to detect longan fruit bunches in longan orchard images.

[0101] It is understandable that the memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. The memory may be an internal storage unit of an electronic device in some embodiments, such as a mobile hard disk of the electronic device. The memory may also be an external storage device of an electronic device in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory may also include both an internal storage unit of the electronic device and an external storage device. The memory may be used not only to store application software and various types of data installed in the electronic device, such as the code of a complex orchard longan string detection program based on image recognition, but also to temporarily store data that has been output or is to be output.

[0102] The processor may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing programs or modules stored in the memory, and calling data stored in the memory.

[0103] Figure 5 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 5The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0104] If the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0105] Embodiment 4:

[0106] This embodiment provides a storage medium, which is a computer-readable storage medium, which stores an Ubuntu 18.04 operating system, a program related to the Python programming language, and a longan image database. When the program related to the Python programming language is executed by a Jetson TX2 GPU processor, the orchard longan fruit bunch detection method of the above embodiment 1 is implemented as follows:

[0107] Constructing an image recognition model; the image recognition model includes a pre-training parameter loading module, an image size normalization module, a target feature extraction module, a pooling module, a multi-scale feature fusion module and a target prediction module; the feature extraction module adopts a MobileNetV3 module, the MobileNetV3 module is the backbone network of the image recognition model, the MobileNetV3 module uses a standard convolution and multiple bneck structures to extract features, the bneck structure combines a deep separable convolution with a residual structure, increases the feature extraction capability of longan fruit clusters, uses a convolution block instead of a fully connected layer after the feature extraction layer, and adds a maximum pooling layer to obtain the final classification result;

[0108] Acquire images of longan fruit bunches in the mature stage in a complex longan orchard scene; perform data amplification and preprocessing on the collected longan fruit bunch images to obtain an initial data set; annotate the longan fruit bunches on each image in the initial data set to obtain an information file in XML format; divide the annotated initial data set into a training set and a validation set according to a set ratio;

[0109] Use the images and annotation data in the training set to train the image recognition model;

[0110] The trained longan fruit bunch detection model is used to detect longan fruit bunches in longan orchard images.

[0111] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0112] In summary, the present invention constructs an image recognition model, which includes a pre-training parameter loading module, an image size normalization module, a target feature extraction module, a pooling module, a multi-scale feature fusion module and a target prediction module, and trains the image recognition model to obtain a longan fruit bunch detection model. The longan fruit bunch detection model can explore the deep-level characteristics of longan fruit bunches and detect more refined longan fruit bunches. The fruit targets, under different lighting and weather conditions, can realize the rapid detection of longan fruit bunches of different varieties, and provide a pair of eyes for the longan fruit bunch picking robot.

[0113] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for detecting longan fruit clusters in a complex orchard based on image recognition, characterized in that: The steps include: Constructing an image recognition model; the image recognition model includes a pre-training parameter loading module, an image size normalization module, a target feature extraction module, a pooling module, a multi-scale feature fusion module and a target prediction module; the feature extraction module adopts a MobileNetV3 module, the MobileNetV3 module is the backbone network of the image recognition model, the MobileNetV3 module uses a standard convolution and multiple bneck structures to extract features, the bneck structure combines a deep separable convolution with a residual structure, increases the feature extraction capability of longan fruit clusters, uses a convolution block instead of a fully connected layer after the feature extraction layer, and adds a maximum pooling layer to obtain the final classification result; Acquire images of longan fruit bunches at maturity in a complex longan orchard scene; Perform data amplification and preprocessing on the collected longan fruit bunch images to obtain the initial data set; Label the longan fruit clusters on each image in the initial data set and obtain an information file in XML format; The annotated initial data set is divided into a training set and a validation set according to the set ratio; Use the images and annotation data in the training set to train the image recognition model; Use the trained longan fruit bunch detection model to detect longan fruit bunches in longan orchard images; The pre-training parameter loading module imports the pre-training parameters and assigns an initial weight value to the target; The image size normalization module adjusts the image to be detected to a uniform pixel size by filling the edges without stretching the image; The pooling module adopts an SPP module, which uses multiple convolution kernels of different sizes to perform maximum pooling processing on the front layer features, and then connects the multiple processed results to form a new feature layer, thereby increasing the network depth while retaining the front layer features to obtain more local feature information; The multi-scale feature fusion module includes a feature pyramid network and a path aggregation network. The path aggregation network first upsamples the result of feature extraction of the MobileNetV3 module, and then downsamples it, so as to enhance the information extraction capability of the feature pyramid network, extract more profound semantic information of longan fruit strings, and detect more refined longan fruit string targets; The method of obtaining the image of a bunch of longan fruits in the mature stage in the complex longan orchard scene is specifically as follows: When the longan fruit is at the ripening stage, choose the daytime on sunny and cloudy days respectively; An RGB camera was used to collect images at a distance of 0.5-1m from the longan fruit; Images were acquired both in direct sunlight and in backlit scenes.

2. The method for detecting longan fruit clusters in a complex orchard based on image recognition according to claim 1, characterized in that: The collected longan fruit bunch images are subjected to data amplification and preprocessing to obtain an initial data set, specifically: Data amplification is performed on the collected longan fruit bunch images, and the image amplification methods include random translation, random cropping and brightness transformation; The collected longan fruit bunch images are preprocessed by image normalization.

3. The method for detecting longan fruit clusters in a complex orchard based on image recognition according to claim 1, characterized in that: The training of the image recognition model using the images and annotation data in the training set specifically includes: Use the model loading module to import pre-trained parameters and assign an initial weight value to the target; Select a predetermined number of images and their annotation data from the training set and input them into the image size normalization module to adjust the image to be detected to a uniform pixel size. When adjusting, the edge is filled in the way without stretching the image. The images adjusted to a uniform pixel size are input into the feature extraction module to extract the deep-level longan fruit cluster features; The feature image after the feature extraction module is operated is subjected to pooling operation and multi-scale feature fusion operation in sequence, and finally the result is predicted; During the model training process, after each round of training, the current image recognition model is verified using the images and annotation data in the verification set, and the weight value of the current image recognition model is adjusted according to the verification results; The maximum number of iterations is set. When the preset maximum number of iterations is reached or the error rate on the validation set no longer decreases, the training is terminated. The image recognition model with the best validation result is selected as the longan fruit string detection model.

4. The method for detecting longan fruit clusters in a complex orchard based on image recognition according to claim 1, characterized in that: The longan fruit bunch detection model trained is used to detect the longan fruit bunches in the longan orchard image, specifically: The longan image collected by the camera in real time is input into the longan fruit bunch detection model, and is processed in sequence by the parameter loading module, image input size normalization module, target feature extraction module, pooling module, and multi-scale feature fusion module of the trained longan fruit bunch detection model. In the prediction module, the maximum circumscribed rectangular boxes of multiple candidate longan fruit bunches are output through convolution operation and the corresponding confidence values ​​are given. The areas marked by all rectangular boxes with confidence values ​​greater than 0.75 are saved to obtain the longan fruit bunch detection results.

5. A complex orchard longan fruit cluster detection system based on image recognition, characterized in that: The complex orchard longan fruit bunch detection method based on image recognition applied to any one of claims 1-4 comprises an image recognition model construction module, a longan fruit bunch image data set acquisition module, a detection model training module, a detection and recognition module and an information display module; The image recognition model construction module includes a pre-training parameter loading module, an image size normalization module, a target feature extraction module, a pooling module, a multi-scale feature fusion module and a target prediction module; the feature extraction module adopts a MobileNetV3 module, which is the backbone network of the image recognition model. The MobileNetV3 module uses a standard convolution and multiple bneck structures to extract features. The bneck structure combines a deep separable convolution with a residual structure, increases the feature extraction capability of longan fruit clusters, uses a convolution block instead of a fully connected layer after the feature extraction layer, and adds a maximum pooling layer to obtain the final classification result; The longan fruit bunch image data set acquisition module is used to collect longan fruit bunch images in direct sunlight and backlight scenes under natural conditions, using a camera during daytime on sunny and cloudy days to form an initial data set of longan fruit bunches, and perform data amplification and preprocessing on the initial data set of longan fruit bunches, annotate the objects in the image, and obtain a training set and a verification set of longan fruit bunches; The detection model training module is used to import the images and annotation data in the longan fruit string training set into the target detection network model for iterative training. After each round of iterative training of the image data in the training set, the images and annotation data in the longan fruit string verification set are imported into the model to verify the performance of the model, and the weight value of the target is adjusted. When the preset maximum number of iterations is reached or the error rate on the verification set no longer decreases, the training is terminated to obtain the optimal longan fruit string detection model; The detection and recognition module is used to load the optimal model weight value, import the longan image collected by the camera in real time into the longan fruit cluster detection and recognition model, and obtain the detection result of the longan fruit cluster; The information display module is used to input the detection result image of the longan fruit cluster into a display device for display.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the complex orchard longan fruit bunch detection method based on image recognition as described in any one of claims 1-4.

7. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the method for detecting longan fruit clusters in a complex orchard based on image recognition as described in any one of claims 1 to 4 is implemented.

Images