A rapid identification and backlight compensation device for fruits grown on trellises

Abstract

The invention relates to a rapid identification and backlight compensation device based on trellis-type cultivated fruits. The device mainly comprises an Intel RealSense D435 depth camera, a machine vision system, a backlight detection and compensation device, a control box, a power module, and a PC. The depth camera is installed at the front end of a robot, the compensation device is installed directly below the depth camera, the power module provides electric energy required for work, the controller is bidirectionally connected to the PC, the depth camera is connected to the PC via a USB cable, the backlight compensation device comprises backlight detection, focus optimization and exposure compensation modules, and the machine vision system adopts a YOLOV4 neural network to perform real-time target detection on fruits.

Description

Technical Field

[0001] The present invention relates to an online detection method and image processing technology for target fruits in the picking work of agricultural robots, and specifically to a method for quickly identifying target fruits in a greenhouse and a backlight compensation device, as well as a method for providing efficient and accurate identification and positioning for agricultural operation robots to perform precise picking in a backlight environment in a greenhouse. Background Art

[0002] With the emerging development of smart agriculture, my country's agricultural digitalization, intelligence, informationization and mechanization technologies have made steady progress. Most of the fruit planting methods in my country, such as grapes, passion fruit, and kiwifruit, are planted in trellis cultivation, and most of the fruits are harvested manually. In the context of the migration of traditional agriculture to intelligent large-scale planting, agricultural intelligent equipment is emerging, and the development of a trellis fruit picking robot is of great significance. The primary key technology of the picking robot is the rapid identification and positioning of fruits, especially in the natural environment, the picking environment is complex and changeable, and backlighting, exposure and fruit occlusion increase the difficulty of fruit picking. In the greenhouse environment, the image acquisition camera uses an upward-looking method to collect the target fruit sample. This situation often occurs due to the backlighting of the image under sunlight or the inability of the structured light camera to obtain the accurate fruit coordinates. Therefore, solving the accurate identification and positioning of the target fruit in the greenhouse environment is a key problem that the picking robot urgently needs to solve. Summary of the invention

[0003] Since the backlight recognition problem during the picking process of agricultural picking robots in greenhouses seriously affects the fruit detection accuracy and rate, in order to overcome the problem of low robot picking efficiency, a rapid recognition method and backlight compensation device based on trellis-cultivated fruits are proposed. The device ensures that clear pictures of the fruits can be taken in a backlight environment.

[0004] A rapid identification method and backlight compensation device for trellis-cultivated fruits, comprising: a depth camera, a machine vision system, a backlight detection and compensation device, a control box, a power module, and a PC. The depth camera is installed at the front end of a picking robot, and the backlight detection and compensation device is installed directly below the depth camera. The power module provides the electric energy required for the work, and the control box and the machine are bidirectionally connected. The backlight detection and compensation device comprises backlight detection, focus optimization and exposure compensation modules, and the machine vision system adopts a YOLOV4 neural network as a target detection network.

[0005] The depth camera model is Intel RealSense D435, which is used to collect RGB images of target fruits in the greenhouse and obtain depth maps.

[0006] The fruit image dataset is used to train the YOLOV4 neural network model and obtain the model weight file. The training stops when the number of iterations is reached, and the model accuracy is adjusted under the effect of the validation set through the loss function during the training process to obtain the best fruit target recognition model for real-time online detection of fruit targets. The training data mainly uses images of fruits collected in the greenhouse using a depth camera, and the collected images are screened and amplified.

[0007] The collected image data were augmented by contrast adjustment and brightness transformation, adding motion and noise interference, and other methods. The amplified images were divided into training set, test set, and validation set in a 6:2:2 ratio.

[0008] The backlight compensation detection searches for the dark area of ​​the image, transfers the metering area of ​​the automatic exposure to the backlight area of ​​the image, exposes the searched dark area, and detects the fruits in the exposure area.

[0009] The current environment image is obtained, and the ratio of the number of low-brightness image blocks to the total number of image blocks in the image is recorded as R_L, and the ratio of the number of high-brightness image blocks to the total number of blocks is recorded as R_H. If the backlight detection condition is met, the current shooting environment is a backlight shooting environment.

[0010] The above-mentioned method for rapid identification of fruits in a greenhouse and backlight compensation device overcome the shortcomings of existing backlight detection and backlight compensation methods for fruits in a greenhouse. Under various backlight conditions, when the fruits are located at different positions in the image, exposure compensation is performed and data sets are collected for neural network training. Finally, the YOLOV4 model is used to perform real-time target detection and positioning of the fruits, which can quickly and accurately obtain the number and position of the fruits in the detection field of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flow chart of fruit recognition and backlight compensation based on trellis cultivation;

[0012] Figure 2 Schematic diagram of the exposure compensation device for simulating agricultural harvesting robots in greenhouses and image acquisition;

[0013] Figure 3 This is a threshold segmentation map of a fruit collected under exposure compensation;

[0014] Figure 4 A schematic diagram of the recognition results using the YOLOV4 neural network under exposure compensation.

[0015] Figure 5 This is the pre-used YOLOV4 neural network architecture diagram. DETAILED DESCRIPTION

[0016] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thoroughly understood.

[0017] A rapid identification method and backlight compensation device based on trellis-cultivated fruits mainly include an Intel Real Sense D435 camera, a machine vision system, a backlight detection and compensation device, a control box, a power module, and a PC. The depth camera is installed at the front end of a robot, the compensation device is installed directly below the depth camera, the power module provides the power required for work, the controller is bidirectionally connected to the PC, the depth camera is connected to the PC via a USB cable, the backlight compensation device includes backlight detection, focus optimization and exposure compensation modules, and the machine vision system uses a YOLO V4 neural network as a target detection network.

[0018] Furthermore, the functions that can be realized by the D435 camera include collecting RGB images of kiwifruit fruits in the greenhouse, obtaining depth maps, etc.

[0019] The target fruit image is used to train the weights of the YOLOV4 neural network to obtain the model file, and the image dataset is used to train the YOLOV4 neural network model. During the training process, when the preset number of iterations is reached or the loss value is minimized and tends to be balanced, the validation set is used to adjust the number of validation training and reconcile the model accuracy to obtain the best target fruit detection model, and then the fruit is identified and detected according to the model.

[0020] The training data mainly uses the images collected by the camera of the fruits in the greenhouse, and the collected images are screened and amplified to avoid the singleness, repetition and invalidity of the images. The collected images are named according to the format of the Pascal VOC dataset, and the Annotations folder is created to store the XML file and the JPEGImages folder is created to store the image and dataset division files.

[0021] Preprocess the dataset, use annotation tools to annotate, set the save path of the XML file, mark the fruit area containing feature points as kiwi, and mark the degree of occlusion and overlap of the target fruit. If the occlusion area is too large or the image exposure area is too small, it will no longer be marked.

[0022] The data set is enlarged, and various amplification processes are performed on the fruit images in the greenhouse by contrast and brightness transformation, motion and noise interference, etc. The amplified images are divided into training set, test set, and validation set in a 6:2:2 ratio.

[0023] Train the YOLOV4 neural network target detection model according to the size of the training set. When the training reaches the preset number of iterations or the loss function reaches the minimum equilibrium, stop training and obtain the model weight file.

[0024] The backlight detection and compensation device can compensate for the required light intensity when the depth camera captures images in a backlight environment. The backlight compensation detection searches for low-brightness areas of the image, transfers the automatic exposure metering area to the backlight area of ​​the image, and exposes the searched dark light area; and detects the fruits in the exposed area.

[0025] Detect whether the current fruit shooting environment is a backlit shooting environment, specifically: obtain the current image to be shot, obtain the ratio of the number of low-brightness image blocks in the image to the total number of image blocks, recorded as R_L, and the ratio of the number of high-brightness image blocks to the total number of image blocks, recorded as R_H. If the backlight detection conditions are met, the current shooting environment is a backlit shooting environment.

[0026] The device automatically detects the backlight compensation, searches for the dark area of ​​the image, transfers the automatic exposure to the low-brightness area of ​​the image, selects the appropriate exposure intensity and time according to the different backlight levels, prevents excessive backlight compensation, and optimizes the focus on the fruit in the dark area to take clear photos of the fruit in the backlight environment, preventing underexposure or overexposure, which causes the neural network detection model to be unable to accurately detect the fruit image.

[0027] The fruit image is divided into m image blocks, the brightness value of each image block is calculated, the brightness value of the root image block is divided into high brightness image blocks and low brightness image blocks, and the following backlight detection conditions are determined:

[0028] The ratio of R_L to R_H is greater than the set backlight detection threshold ThresholdA.

[0029] Obtain the brightness value of each image block and convert it into the brightness evaluation value of the image. According to the brightness evaluation mapping table, obtain the backlight degree of the image; if the backlight detection condition is met, it is determined that the current fruit is in the backlight detection environment. After detecting the backlight, the backlight image can be accurately raised to a suitable brightness through backlight compensation, and the compensated state is locked. The camera acquires the fruit image from the bottom from near to far. The fruit recognition method in the greenhouse under the backlight environment uses the neural network YOLOV4 algorithm to predict the global information of the image, and constructs the target fruit detection model. First, the fruit image is recognized, and then the target fruit detection features are extracted.

[0030] The system detects and returns to normal state, waiting for the next fruit identification.

[0031] The target fruit image is divided into m image blocks, the brightness value of each image block is calculated, and the image blocks are divided into high-brightness image blocks and low-brightness image blocks according to their brightness values. It is determined whether the following backlight compensation termination detection conditions are met:

[0032] The ratio of R_L to R_H is less than the threshold ThresholdA;

[0033] If the backlight compensation termination condition is met, the backlight compensation should be terminated.

[0034] YOLOV4 neural network uses CSPDarknet 53 convolutional neural network to extract target features, introduces SPP network structure, and uses three scale tensor prediction outputs. It has good detection effect in the case of different target sizes and partial occlusion. The neural network predicts 4 coordinates t x ,t y ,t w ,t h , the leftmost offset coordinate of the target in the image (c x , c y ) and the height and width of the bounding box are p w 、p h The output prediction value includes b x , b y , b w , b h And 5 confidence levels. The confidence level is the credibility of the prediction result. The prediction value is calculated as follows:

[0035] b x =σ(t x )+c x

[0036] b y =σ(t y )+c y

[0037]

[0038]

[0039] Use YOLOV4 to identify the target fruit, and the rectangular box output by the predicted value is the actual target area.

[0040] The original camera image is passed into the network input layer, and the CBM module performs convolution operations, followed by batch normalization and activation function activation. The next step is to pass it into the CSP structure to complete feature extraction; then pass it into the YOLOV4 structure for three scale predictions and regressions (19×19, 32×32, 76×76), and then regress to the original image for multi-scale target prediction. The three-scale FeatureMap is integrated to complete target detection, and the original image resolution is bx , b y The pixel is the center of the area, and the width is b x , high b y The target is extracted from the pixel area, and finally the fruit area image is obtained, that is, the 2D coordinate area tensor of the picking target in the original image.

[0041] The specific implementation process is as follows. Turn on the robot vision camera and control the distance from the camera installation position to the fruit within 500mm: the PC sends a command, and the camera starts to collect images of the fruits in the greenhouse; the system processes the collected images and calculates the ratio of the number of high-brightness image blocks to the number of low-brightness image blocks to the total number of image blocks; judges according to the backlight detection conditions, and when it is in a backlight state, transfers the exposure area to the low-brightness area of ​​the image to fill in the light for the target fruit; the camera acquires the image and judges according to the backlight detection conditions; when it is in a backlight state, continues to perform backlight compensation; when it is not in a backlight state, uses the trained YOLOV4 target fruit detection model to detect the target fruit; judges the current number of fruits to provide a basis for picking for the picking robot.

[0042] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A rapid identification and backlight compensation device based on trellis-type cultivated fruits, comprising: A depth camera, a machine vision system, a backlight detection and compensation device, a control box, a power module, and a PC, wherein the depth camera is installed at the front end of the picking robot, the backlight detection and compensation device is installed directly below the depth camera, the power module provides the power required for the work, the controller is bidirectionally connected to the PC, the backlight detection and compensation device includes backlight detection and exposure compensation, and the machine vision system uses the YOLOV4 neural network as the target detection network; The backlight compensation detection system searches for low-brightness areas in the image, shifts the automatic exposure area to the backlight area, and adjusts the exposure of the searched low-brightness areas; it adjusts the brightness of the exposure area to a lower level; Get the current captured image, extract the ratio of the number of low-brightness image blocks to the total number of image blocks in the image, record it as R_L, and the ratio of the number of high-brightness image blocks to the total number of image blocks, record it as R_H. If the backlight detection condition is met, the current shooting environment is a backlight shooting environment; Method for using the rapid identification and backlight compensation device based on trellis-type cultivated fruits: The target fruit image is divided into m image blocks, the brightness value of each image block is calculated, the brightness value of the root image block is divided into high brightness image blocks and low brightness image blocks, and the following backlight detection conditions are determined: The ratio of R_L to R_H is greater than the preset weight value ThresholdA, where ThresHoldA is the set backlight detection threshold, which is set according to the optimal light intensity for different fruit identification; The brightness value of each image block is obtained and converted into the brightness value of the image to obtain the backlight degree of the image; if the backlight detection condition is met, it is determined that the current target fruit is in the backlight detection environment; after detecting the backlight, the backlight image can be accurately raised to a suitable brightness through backlight compensation, the compensated state is locked, the system detects and returns to the normal state, and waits for the next target fruit recognition; The target fruit image is divided into m image blocks, the brightness value of each image block is calculated, and the image blocks are divided into high-brightness image blocks and low-brightness image blocks according to their brightness values. It is determined whether the following backlight compensation termination detection conditions are met: The ratio of R_L to R_H is less than the preset threshold ThresholdA; If the backlight compensation termination condition is met, the backlight compensation should be terminated.

2. The device according to claim 1, characterized in that The depth camera model is Intel Real Sense D435, which is used to collect RGB images and obtain depth maps of fruits in the greenhouse.

3. The device according to claim 1, characterized in that The fruit image dataset is used to train the YOLOV4 neural network model to obtain the model weight file. The training is stopped when the number of iterations is reached. During the training process, the model accuracy is adjusted under the action of the validation set through the loss function to obtain the optimal fruit target detection model for real-time online detection of fruit targets.

4. The device according to claim 3, characterized in that The network training data set is an image of the target fruit in the greenhouse captured by a depth camera, and the data is screened and amplified.

5. The device according to claim 4, characterized in that The collected image data augmentation includes adjusting the image contrast and brightness, adding motion and noise interference methods, and dividing the augmented images into training set, test set, and validation set in a 6:2:2 ratio.

Images