A target detection method based on multi-modal data fusion and an in-body fruit picking method based on a target detection model

Abstract

The application discloses a target detection method based on multi-modal data fusion and an in-vivo fruit picking method based on a target detection model, and belongs to the technical field of intelligent detection. In the training method of the target detection model, multi-modal visual data of in-vivo fruits in a natural environment is acquired by using a depth camera, a coordinate attention mechanism is introduced to enhance the perception ability of a feature extraction network to target objects, and a depth separable convolution module is combined to reduce the model parameter quantity and inference time. On the other hand, the target detection model is applied to the recognition of fruits by a mechanical arm, a visual servo detection mechanism based on fruit position information and a shielding state classification is proposed, the mechanism takes advantage of the motion characteristics of the mechanical arm in the fruit picking process, changes the camera view angle through the movement of the mechanical arm, constantly updates the fruit targets detected in the camera field of view, realizes dynamic detection of the fruits, overcomes the missed detection caused by light and fruit shielding, and improves the detection rate of the fruits.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent detection, and particularly relates to a target detection method based on multi-modal data fusion and a fruit picking method in vivo based on a target detection model. BACKGROUND

[0002] With the gradual improvement of the modernization degree of agricultural production, more and more traditional manual labor can be replaced by machines. Selective fruit picking operation is the most time-consuming, labor-intensive and seasonal link in the agricultural production chain, so improving the automation degree of the picking link will greatly improve the efficiency of agricultural production. Automatic picking will inevitably involve detection and positioning of the fruit and other pickable parts on the crops. However, in the unstructured environment of the orchard, the visibility of the fruit is generally low due to the shading and covering of the branches and leaves of the fruit trees, which limits the operation of the agricultural robot. Therefore, it has important application value to accurately and quickly detect the occluded fruit in a variable light and unstructured environment.

[0003] In recent years, with the combination of artificial intelligence and agricultural informatization, the application scenarios based on computer vision technology have been expanded to agricultural production. Benefiting from the low difficulty of deploying machine vision technology, simple platform construction and low cost, many farms have begun to apply it in practice. For example, Kurtser and Edan proposed in the IEEE International Workshop on Intelligent Robots and Systems 2018 to use a dynamic perception strategy to improve the detection performance of a sweet pepper picking robot. The study determines whether the current perspective needs an additional detection viewpoint through a monocular vision algorithm, and the selection of the additional viewpoint is based on the degree of occlusion of the fruit and the expected number of fruit targets under that perspective, and the best viewpoint is selected from the preset positions according to the efficiency of performing this operation. However, monocular vision algorithms are difficult to make accurate decisions in complex lighting environments, and cannot detect fruits that are completely covered by branches and leaves. Lehnert et al. proposed in the IEEE International Workshop on Intelligent Robots and Systems 2019 to use a 3D camera array combined with an end effector to obtain multiple perspectives from different angles to find the next optimal viewpoint. It uses the captured multiple perspectives and fruit segmentation results to estimate the gradient of the best perspective direction. However, the images obtained by the 3D camera array under different perspectives need to be processed and calculated respectively, which cannot meet the real-time requirements of fruit automatic picking. At the same time, using RGB images as the only information for detecting fruits in the complex environment of the orchard is prone to changes in fruit features due to different fruit maturity, varieties and variable lighting conditions in the orchard, which cannot achieve accurate and robust fruit detection.

[0004] From the above-mentioned technologies, the existing fruit detection method still has defects such as limited detection angle, weak perception ability and poor anti-interference performance, and the visual servo technology combined with multi-modal visual data and multi-view detection can reduce the influence of environmental factors on the detection effect to a certain extent, so how to combine the multi-modal visual data and the visual servo technology of multi-view detection is also the key problem for the person skilled in the art to study. SUMMARY

[0005] In view of the problems that the existing detection model has weak target perception ability and poor recognition ability, the application provides a target detection method based on multi-modal data fusion, introduces a coordinate attention mechanism to enhance the perception ability of a feature extraction network to target objects, combines a depth separable convolution module to reduce the model parameter quantity and inference time, so that the improved model has less calculation amount and parameter quantity, smaller model volume and faster detection speed, and a good balance is achieved in model detection resource occupation and detection speed, and is suitable for deployment and application on an embedded platform.

[0006] In addition, in view of the problem that in an unstructured orchard, the disordered growth of fruit trees causes the fruits to be completely covered by dense branches and leaves, and the fruits cannot be detected in a single view, the application further provides an in-body fruit picking method based on the above target detection model. The method proposes a visual servo mechanism for selective picking by using the position information and the occlusion state of the fruits to guide the mechanical arm, updates the view of the camera during the movement of the mechanical arm to form a view set, finds the view with the maximum pickability of the fruits in the view set, and records the position of the mechanical arm when the view is recorded as the starting point of the next picking task. The process is repeated until there is no directly pickable target in the field of view of the camera, thereby improving the fruit detection rate.

[0007] To achieve the purpose of the application, the following technical solutions are adopted:

[0008] The first aspect of the application provides a target detection method based on multi-modal data fusion, comprising the following steps:

[0009] S110: constructing a target detection model based on multi-modal data fusion, the target detection model comprising a feature extraction network, a neck structure and a prediction layer, and introducing a coordinate attention mechanism in the feature extraction network and the neck structure to enhance the perception ability to target objects, wherein the multi-modal visual data comprises multi-modal visual data of RGB images, infrared images and depth images;

[0010] S120: obtaining multi-modal visual data of in-body target objects as a training set, and inputting the target detection model for training to obtain a trained target detection model;

[0011] S130: Acquire the multimodal visual data to be tested, which includes the target object to be picked; input the multimodal visual data to be tested into the target detection model to identify the target object in the multimodal visual data, and output the identification result.

[0012] As an example, introducing coordinate attention mechanisms in both the feature extraction network and the neck structure to enhance the ability to perceive targets includes: embedding a coordinate attention module in the feature extraction network and embedding a coordinate attention module at the end of the neck structure.

[0013] As an example, the feature extraction network adopts the YOLOv5s model. In the YOLOv5s model, depthwise separable convolutions replace ordinary convolutions in the CBL module to form the DPBL module. After the CSP_1_X module is cascaded, a coordinate attention module is embedded to form the CSP_CA_X module.

[0014] The neck structure includes a DPBL module and a CSP_2_X module. The CSP_2_X module is composed of a cascade of ordinary convolution and X Resunits. A coordinate attention module is embedded at the end of the neck structure.

[0015] The number of channels in the Focus module structure of the feature extraction network was adjusted to five to enable the reading of multimodal visual data.

[0016] As an example, the coordinate attention module is specifically used for:

[0017] For the input feature map Each channel is encoded using pooling kernels of size H×1 and 1×W along the horizontal and vertical coordinates; therefore, the height is... h The c Each channel and width is w The c The outputs of each channel are represented as shown in equations (1) and (2), respectively. For feature map exist The value of the position, For feature map exist The value of the position.

[0018] (1)

[0019] (2)

[0020] The two transformations described above are aggregated along two spatial directions respectively, and then cascaded to generate two feature maps. and a shared 1x1 convolution kernel is used for convolution operation .

[0021] As formula (3), the generated is the intermediate feature map that encodes spatial information in the horizontal and vertical directions, is a nonlinear activation function.

[0022] (3)

[0023] Then, along the spatial dimension, the f is divided into two separate tensors and , combined with two convolution kernels with a size of 1x1 and transform the feature map and to the same number of channels as the input X, generating a pair of direction-aware and position-sensitive attention maps and , respectively as formula (4), (5) shown, wherein is a Sigmoid activation function.

[0024] (4)

[0025] (5)

[0026] Finally, the and act on the feature map X to obtain a coordinate attention map with the same size as the feature map X, and the calculation formula (6) is as follows:

[0027] (6)

[0028] As an example, the step S120 includes:

[0029] Configure the target detection model training key parameters, including the size of the input image, the number of categories, the number of training iterations, the initial learning rate, the learning rate adjustment strategy and the number of input images per batch;

[0030] CIOU Loss is used as the bounding box loss function, and the target detection model is trained to converge. The weight file obtained after each training iteration is saved, and the weight file with the best validation effect is saved, thereby obtaining the trained target detection model.

[0031] As an example, the step of obtaining the multi-modal visual data of the in-vivo target object as the training set includes:

[0032] acquiring target object image information, the image information including multi-modal visual data of RGB images, infrared images and depth images;

[0033] classifying and marking the target object in the picking environment image according to the occlusion condition, the classification and marking including three categories of non-occlusion, occlusion by non-influencing objects and occlusion by influencing objects;

[0034] dividing the obtained multi-modal visual data of the target object and the classification and marking into a training set, a validation set and a test set according to a predetermined ratio.

[0035] As an example, the feature extraction network adopts an improved CSPDarknet53 network; the CSPDarknet53 includes 10 modules {Focus, CBL, CSP_1_X, CBL, CSP_1_X, CBL, CSP_1_X, CBL, CSP_1_X, SPP}; wherein the Focus module adopts a slicing operation to downsample the target object image information; the CBL module includes three parts of ordinary convolution, batch normalization processing and an activation function; the CSP_1_X module is composed of a CBL module, X Res units, ordinary convolution and a Concat operation, the Res unit is a network structure constructed by referring to the residual structure idea, and is composed of the initial input and the output after two consecutive CBL modules; the SPP (Spatial Pyramid Pooling) module adopts four different size pooling kernels to perform a pooling operation on the input feature map, and then concatenates different pooling results to realize the fusion of multi-scale features.

[0036] The second aspect of the present application provides a fruit picking method in the body based on a target detection model, including the following steps:

[0037] S210: obtaining an identification result according to the above-mentioned target detection method based on multi-modal data fusion, the identification result including multiple classification marks and boundary box coordinates of the target object, and the classification marks including three categories of non-occlusion, occlusion by non-influencing objects and occlusion by influencing objects;

[0038] S220: obtaining a first spatial position coordinate of the boundary box coordinate of the target object relative to the depth camera; and converting the spatial position coordinate of the target object relative to the depth camera into mechanical arm base position coordinate information;

[0039] S230: Obtain the original mechanical arm position information, control the mechanical arm to move to the specified first spatial position, and perform picking work on the target object; randomly select one classification label as unobstructed or obstructed by a non-impact object as a pickable target fruit to perform picking task; calculate the next motion starting point of the mechanical arm according to the identification result and positioning result of the target object in multiple groups of picking environment images combined with the corresponding mechanical arm position information.

[0040] S240: Repeat the above S210 to S230 until there is no pickable object in the field of view of the depth camera carried by the mechanical arm, and the picking work is completed.

[0041] As an example, the step S210 further comprises:

[0042] Receiving multi-modal visual data of picking environment images taken by the depth camera in real time, the multi-modal visual data including RGB images, depth images and infrared images;

[0043] Channel fusion is performed on the RGB images, depth images and infrared images, and the image size is adjusted to a predetermined size to obtain a multi-modal image;

[0044] The multi-modal image is input into the target detection model to output the boundary box coordinates and classification label corresponding to the target object and the corresponding confidence;

[0045] According to the confidence threshold judgment, the prediction results with small confidence are removed, and the boundary box coordinates and class probability of the target fruit are obtained;

[0046] The non-maximum suppression algorithm is used to remove redundant boundary boxes on the same target object to obtain the best prediction result.

[0047] As an example, the step S240 further comprises:

[0048] During picking, the depth camera is controlled to obtain multiple groups of multi-modal images of the target on the motion path, and the mechanical arm position information of each group of multi-modal images of the target is recorded;

[0049] The multi-modal image set of multiple groups of targets on the motion path of the mechanical arm is represented as follows, v 1 to v n The multi-modal image of the target fruit tree from the first group to the nth group is represented as follows:

[0050]

[0051] The multi-modal images of multiple groups of targets are sequentially input into the target detection model to output multiple classification labels and boundary box coordinates corresponding to the target object;

[0052] setting the unobstructed and the obstructed by non-influencing objects as pickable objects (PO) and setting the obstructed by influencing objects as unpickable objects (NPO);

[0053] calculating the number of pickable objects (PO) and unpickable objects (NPO) in each group of multi-modal images, and calculating the pickability of the target fruit in each group of multi-modal images by the following formula

[0054]

[0055] calculating the multi-modal image group with the highest pickability of the target fruit , and the calculation formula is as follows:

[0056]

[0057] The corresponding mechanical arm position information of the multi-modal image group is used as the next motion starting point of the mechanical arm, and the mechanical arm is controlled to move to the motion starting point when performing the next picking task.

[0058] The third aspect of the present application provides a fruit picking system based on a target detection model, comprising:

[0059] An image processing module is used to obtain multi-modal visual data shot by a depth camera, and the multi-modal visual data contains target objects to be picked; a target detection model trained by using a target detection method based on multi-modal data fusion is used to identify and locate the target objects in the multi-modal visual data, and output multiple classification labels and boundary box coordinates of the target objects, wherein the classification labels include unobstructed, obstructed by non-influencing objects and obstructed by influencing objects;

[0060] A spatial coordinate conversion module is used to obtain first spatial position coordinates of the boundary box coordinates of the target objects relative to the depth camera; and convert the spatial position coordinates of the target objects relative to the depth camera into mechanical arm base position coordinate information;

[0061] A motion control module is used to obtain the original mechanical arm position information, control the mechanical arm to move to a specified first spatial position, and perform picking work on the target objects; randomly select a classification label as a pickable target fruit, and execute a picking task;

[0062] A motion calculation module is used to calculate a next motion starting point of the mechanical arm according to the identification results and positioning results of the target objects in multiple groups of picking environment images, and in combination with corresponding mechanical arm position information.

[0063] The fourth aspect of the present application provides an electronic device, comprising a processor, an input device, an output device and a memory, which are connected in sequence, the memory is used for storing a computer program, the computer program comprises program instructions, and the processor is configured to invoke the program instructions to execute the above-mentioned method.

[0064] The fifth aspect of the present application provides a readable storage medium, which stores a computer program, the computer program comprises program instructions, and the program instructions make the processor execute the above-mentioned method when the processor executes the program instructions.

[0065] Compared with the prior art, the beneficial effects of the present application are:

[0066] (1) In the target detection model design process of the present application, a depth separable convolution is used instead of a normal convolution operation; the improved model has less calculation amount and parameter amount, smaller model volume and faster detection speed, and a good balance is achieved in model detection resource occupation and detection speed, which is suitable for deployment and application on an embedded platform, and meets the real-time detection requirements of a real picking robot.

[0067] (2) The coordinate attention mechanism is embedded in the feature extraction layer and the neck structure of the target detection model of the present application, the attention mechanism can enhance the attention of the network to the target feature information and suppress other useless background features, so as to improve the accuracy of the prediction result of the model; considering that most attention mechanisms will bring additional calculation overhead to a lightweight network, the coordinate attention mechanism introduced in the present application is simple and flexible, and can efficiently improve the detection precision and accuracy of the network.

[0068] (3) In the model training process of the present application, the bounding box prediction standard of YOLOv5s is redefined, and CIoU is used to more accurately represent the overlap degree of two bounding boxes; CIoU measures the aspect ratio of the anchor and the aspect ratio of the target box with a proportion coefficient, so that the present application has faster and more accurate regression in the training process, and also makes the detection algorithm more friendly to small targets in the image.

[0069] (4) The present application fuses RGB images, depth images and infrared images in the channel direction to obtain multi-modal visual data; the infrared image can provide information for the target detection task in the case of weak light and at night. The depth image is not affected by brightness and color changes and can also provide useful information in complex scenes, and the target detection model of the present application can process multi-modal information algorithm and network at the same time, which can improve the target recognition accuracy in complex lighting environments.

[0070] (5) The application adopts the "eye in hand" calibration mode for the depth camera and the mechanical arm, and controls the depth camera to collect images of target fruits at different angles during the movement of the mechanical arm. In order to prevent the end effector of the mechanical arm from causing damage to the mechanical arm and the fruit tree during forced picking, the fruits blocked by branches and other fruits are defined as un-pickable objects in the application; the continuous change of the angle can make the picking robot find more pickable objects, prevent overlapping fruits and fruits blocked by branches from being identified and picked; in addition, the target detection model described in the application identifies the position and blocked form of the fruits in the view, calculates the pickable rate of the fruits in all views according to the identification result, finds the view with the highest fruit picking rate, and records the position of the mechanical arm in this view as the starting point of the next movement of the mechanical arm, which is convenient and efficient. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 A target detection method flowchart based on multi-modal data fusion is provided for the embodiment of the application.

[0072] Figure 2 A fruit detection and picking method flowchart is provided for the embodiment of the application Figure 1 ;

[0073] Figure 3 A fruit picking method flowchart based on a target detection model is provided for the embodiment of the application Figure 2 ;

[0074] Figure 4 A block diagram of an electronic device is provided for the embodiment of the application.

[0075] Figure 5 A deep separable convolution module structure diagram is provided for the embodiment of the application.

[0076] Figure 6 A coordinate attention module structure diagram is provided for the embodiment of the application.

[0077] Figure 7 A target detection model structure diagram is provided for the embodiment of the application.

[0078] Figure 8 A fruit picking system block diagram based on a target detection model is provided for the embodiment of the application. DETAILED DESCRIPTION

[0079] Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein.

[0080] Exemplary method

[0081] In the complex environment of orchard, the current detection model has weak target perception ability and poor recognition ability. Figure 1 and Figure 2 The example provides a target detection method based on multi-modal data fusion, including the following steps:

[0082] S110: Construct a target detection model based on multi-modal data fusion, which includes a feature extraction network, a neck structure and a prediction layer. The coordinate attention mechanism is introduced in the feature extraction network and the neck structure to enhance the target perception ability. The multi-modal visual data includes RGB images, infrared images and depth images.

[0083] Specifically, as shown in Figure 7 The example constructs a target detection model based on multi-modal data fusion by improving the YOLOv5s model. The target detection model based on improved YOLOv5s includes a feature extraction network, a neck structure and a prediction layer. The coordinate attention module is embedded in the feature extraction network. The coordinate attention module is embedded at the end of the neck structure, which can improve the feature extraction ability of the target detection model, highlight the feature information of the target, suppress the interference of complex background factors, and reduce the parameter amount of the model.

[0084] As a possible implementation, the feature extraction network uses an improved CSPDarknet53 network. CSPDarknet53 includes 10 modules {Focus, CBL, CSP_1_X, CBL, CSP_1_X, CBL, CSP_1_X, CBL, CSP_1_X, SPP}. The Focus module uses slicing operation to downsample the image. The CBL (Convolution, Batch Normalization, and Leaky-ReLU) module includes three parts: ordinary convolution, batch normalization processing and activation function. The CSP_1_X module is composed of a CBL module, X Res units, an ordinary convolution, and a Concat operation. The Res unit is a network structure constructed by referring to the residual structure idea, which is composed of the initial input feature map and the feature map after two consecutive CBL modules. The SPP (Spatial Pyramid Pooling) module uses four different size pooling kernels to perform pooling operation on the input feature map, and then concatenates the different pooling results to realize the fusion of multi-scale features.

[0085] As preferred, the number of channels of the first layer Focus module structure in the feature extraction network is adjusted to five channels to realize reading of multi-modal visual data.

[0086] It should be noted that the improved CSPDarknet53 network in this example introduces a DPBL (Depth-wise Convolution, Point-wise convolution, Batch Normalization, and Leaky-ReLU) module to replace the CBL module, and the DPBL module uses a depth separable convolution to replace the ordinary convolution in the CBL module. The depth separable convolution can effectively reduce the parameter quantity of the model and the overall calculation quantity of the network compared with the ordinary convolution. The improved CSP_CA_X module embeds a coordinate attention module CA (Coordinate Attention) after the Concat operation of the CSP_1_X module.

[0087] In one example, the neck structure of the YOLOv5s model includes four CBL modules and four CSP_2_X modules, and the CSP_2_X module is composed of an ordinary convolution and X Res unit cascades. The target detection model replaces the CBL module in the neck structure with a DPBL module and embeds a coordinate attention module CA at the end of the neck structure.

[0088] Specifically, as shown in Figure 5 , the depth separable convolution module in this example is composed of a depth convolution and a point convolution. For the input feature map X, assuming that the feature map size is , the depth separable convolution uses a depth convolution kernel with a size of to convolve each channel, and then uses a point convolution with a size of to weight and combine the output feature map in the depth direction, adjust the channel number of the feature map to , and finally output a feature map with a size of .

[0089] S120: Obtain multi-modal visual data of the in-vivo target object as a training set, and input the target detection model for training to obtain a trained target detection model.

[0090] Specifically, the network model training key parameters are pre-set, the key parameters include the size of the input image (size=640x640x5), the number of categories (num_class=4), the number of training iterations (epoch=200), the initial learning rate (learning_rate=0.001), the learning rate adjustment strategy (decrease by 0.0005 every 10 epochs), and the number of images processed per batch (batch_size=8).

[0091] CIOU is adopted Loss As a bounding box loss function, the target detection model is trained to converge, the weight file obtained after each training iteration is saved, the weight file with the best verification effect is saved, and thus the trained target detection model is obtained.

[0092] During the training process, the loss value change of the validation set after each iteration of the model is recorded, if the loss value tends to zero when the number of training iterations reaches 200, and the average accuracy of the test set reaches 0.8 under the condition that the IoU (intersection over union) is set to 0.5, the target detection model is obtained by continuing to train to the maximum number of iterations; otherwise, the training model is terminated, and the network model training key parameters are reconfigured, the reconfigured key parameters including the initial learning rate (learning_rate) and the learning rate adjustment strategy.

[0093] The present example enhances the perception ability of the feature extraction network to the target object by introducing the coordinate attention mechanism, and reduces the model parameter quantity and the inference time by combining the depth separable convolution module, so that the improved model has less calculation amount and parameter quantity, smaller model volume and faster detection speed, and a good balance is achieved in model detection resource occupation and detection speed, which is suitable for deployment and application on an embedded platform.

[0094] It should be noted that the example of the present example is taken as an example of fruit target in an orchard, but it should be understood that tea target recognition in a tea garden, vegetable recognition in a vegetable garden and other application scenarios requiring target detection can also use the method of the present example, and the above scenarios are all included in the protection scope of the present application.

[0095] As Figure 6 shown, in a possible implementation, the coordinate attention module processing flow is specifically as follows:

[0096] For the input feature map , a pooling kernel with a size of Hx1 and 1xW is used to encode each channel along the horizontal coordinate and vertical coordinate directions, so that the height of the first h channel is c and the width of the first w channel is cThe outputs of the two channels represent respectively as shown in formula (1), (2):

[0097] (1)

[0098] (2)

[0099] The above two transformations are aggregated along two spatial directions respectively, and then concatenated to generate two feature maps and , and then a shared 1x1 convolution kernel is used for convolution operation ; wherein is the value of the feature map at the position, is the value of the feature map at the position.

[0100] As formula (3), the generated is an intermediate feature map that encodes spatial information in horizontal and vertical directions, is a nonlinear activation function.

[0101] (3)

[0102] Then, along the spatial dimension, the f is divided into two separate tensors and , combined with two convolution kernels with a size of 1x1 and transform the feature maps and to the same number of channels as the input X, generating a pair of direction-aware and position-sensitive attention maps and , calculated as shown in formula (4), (5) respectively, wherein is a Sigmoid activation function.

[0103] (4)

[0104] (5)

[0105] Finally, the and act on the feature map X to obtain a coordinate attention map with the same size as the feature map X, calculated as formula (6):

[0106] (6)

[0107] As an example, the prediction layer of the target detection model adopts CIOU Loss As a bounding box loss function, the CIOU Loss The function is shown in equation (7), and The values are shown in equations (8), (9):

[0108] (7)

[0109] (8)

[0110] (9)

[0111] wherein, represents the coordinates of the center point of the predicted box, represents the coordinates of the center point of the real box, and represent the width and height of the predicted box, respectively, and represent the width and height of the real box, represents the Euclidean distance between the center points of the predicted box and the real box, represents the diagonal distance of the smallest closed region that can contain both the predicted box and the real box.

[0112] S130: Obtain the to-be-tested multi-modal visual data, which contains the target object to be picked; input the to-be-tested multi-modal visual data into the target detection model to identify the target object in the multi-modal visual data, and output the identification result.

[0113] Specifically, the to-be-tested multi-modal visual data uses the multi-modal visual data captured by the depth camera, inputs the to-be-tested multi-modal visual data into the target detection model to identify and locate the target object in the multi-modal visual data, and outputs the identification result. The identification result includes multiple classification labels and bounding box coordinates of the target object, and the classification labels include three types of unobstructed, obstructed by non-influencing objects, and obstructed by influencing objects.

[0114] As an example, the step of obtaining multi-modal visual data of in-vivo targets includes:

[0115] S121: Collect target object image information, which includes multi-modal visual data of RGB images, infrared images, and depth images.

[0116] Specifically, the collection target here refers to image collection of mature fruit picking environment using a depth camera. In the implementation process, the image collection time is distributed in different time periods such as morning, noon, afternoon, and evening. The collected images can include normal light images, strong light images, backlight images, and night artificial light images. The image information here is multi-modal visual data including RGB images, infrared images, and depth images.

[0117] As a possible implementation, the present example uses a depth camera to collect image data of peach picking environment, and the collection time includes 8:00-11:00 am, 12:00-1:00 pm, 2:00-5:00 pm, and 6:00-9:00 pm. First, fix the depth camera using a tripod, keep the camera lens view angle horizontal, and perpendicular to the peach tree trunk, then adjust the tripod height to make the camera distance from the peach tree trunk about 0.5-1.5 m, adjust the tripod height to 1.1-1.8 m, set the elevation angle to 15-30°, rotate the depth camera horizontally during shooting, and control the rotation angle to-30-30°. The multi-modal visual data obtained by shooting is saved in mkv video format. Then use the ffmpeg video processing tool to extract the RGB image, depth image, infrared image, and corresponding timestamp camera internal and external parameters at a rate of 3 frames per second from a total of 175 videos, and obtain not less than 3000 groups of visual data under different environments such as normal light, strong light, backlight, dark light, and night.

[0118] S122: uniformly name each group of multi-modal visual data.

[0119] Preferably, in order to facilitate the model to accurately read each group of multi-modal data, each group of multi-modal visual data is renamed, and the naming format consists of four consecutive digits of data category and number. The RGB image is named rgb0001.png, rgb0002.png, …, rgb2050.png, the depth image is named depth0001.png, depth0002.png, …, depth2050.png, and the infrared image is named ir0001.png, ir0002.png, …, ir2050.png.

[0120] S123: classify and label the picking environment image (target object image information), for example, the picking environment image can be labeled using LabelImg software, and the classification label includes three categories of unobstructed, obstructed by non-influencing objects, and obstructed by influencing objects according to the occlusion of fruits in the picking environment.

[0121] It should be understood that the picking environment image herein refers to the image of all objects covered by the depth camera when shooting, and the target object image information is only a part of the picking environment image. For example, in the process of picking peaches, the depth camera will record objects including the trunk, leaves and peaches of the peach tree, and the target object image information herein refers only to the peaches as the picking target image information.

[0122] Specifically, in the picking process, the classification label is set as not being blocked (NO), being blocked by non-influencing objects (OL), being blocked by influencing objects (OF), and being blocked by the trunk (OB). In order to avoid damage to the robot arm and the fruit tree caused by the robot arm end effector forcibly picking the fruit during the picking process, the not being blocked (NO) and the being blocked by leaves (OL) are set as pickable objects (PO), and the being blocked by other fruits (OF) and the being blocked by the trunk (OB) are set as non-pickable objects (NPO).

[0123] S124: The obtained target object image information and classification label are randomly divided into a training set, a validation set and a test set according to a predetermined ratio.

[0124] Specifically, the obtained target object image and label information are randomly divided into a training set, a validation set and a test set in a ratio of 7:1:2. The ratio can be adjusted according to actual conditions, which is not limited to the present application.

[0125] Referring to FIGS. 1 and 2, Figure 2 and Figure 3 The present example also provides a method for picking fruits in vivo based on a target detection model, comprising the following steps:

[0126] S210: obtaining an identification result according to the above-mentioned method for target detection based on multi-modal data fusion, wherein the identification result comprises multiple classification labels and boundary box coordinates of the target object, and the classification labels comprise three types of not being blocked, being blocked by non-influencing objects and being blocked by influencing objects.

[0127] The specific implementation steps are as follows:

[0128] S211: receiving multi-modal visual data of a picking environment image shot in real time by a depth camera, wherein the multi-modal visual data comprises an RGB image, a depth image and an infrared image;

[0129] S212: performing channel fusion on the RGB image, the depth image and the infrared image, and setting the image size to 640x640x5 through operations such as adjusting the aspect ratio and scaling, to obtain a multi-modal image.

[0130] S213: inputting the multi-modal image into the target detection model to output the boundary box coordinates and the classification label corresponding to the target object and the corresponding confidence.

[0131] Specifically, the multi-modal image with the size of 640x640x5 is input into the target detection model, and three feature maps with the sizes of 3x80x80 (box+confidence+class_num), 3x40x40 (box+confidence+class_num) and 3x20x20 (box+confidence+class_num) are output. Wherein, the box is the bounding box coordinate (x, y, w, h) of the predicted target, which respectively represents the center point coordinate and the length and width value of the predicted target box; the confidence is the confidence; the value of class_num is 4, which here indicates the probability of four categories of unobstructed (NO), obstructed by leaves (OL), obstructed by other fruits (OF) and obstructed by tree trunk (OB).

[0132] S215: By confidence threshold judgment, remove the prediction result with confidence less than the confidence threshold, and obtain the bounding box coordinate, confidence and category probability of the target fruit.

[0133] S216: By non-maximum suppression algorithm, delete the redundant detection box on the same target, and obtain the best prediction result.

[0134] S220: Obtain the first spatial position coordinate of the target object relative to the depth camera; and convert the spatial position coordinate of the target object relative to the depth camera into the base position coordinate information of the robot arm.

[0135] Specifically, the position of the depth camera relative to the robot arm adopts a hand-eye calibration method, that is, the camera is installed on the end effector of the robot arm; the relative position relationship X between the depth camera and the end effector of the robot arm is calculated, and the solving method is as shown in S221-S227.

[0136] S221, establish a hand-eye calibration system, the hand-eye calibration system includes a robot arm, a calibration board, an end effector and a depth camera, here the architecture of the hand-eye calibration system is common, and will not be repeated here.

[0137] S222, according to the relationship matrix of the robot arm base to the calibration board Invariable, formula (10) is established. Wherein T represents the conversion matrix of the end effector coordinate system to the robot arm base coordinate system, T represents the conversion matrix of the camera coordinate system to the end effector coordinate system, T represents the conversion matrix of the calibration board coordinate system to the camera coordinate system.

[0138] (10)

[0139] S223, transform the pose of the mechanical arm, then:

[0140] (11)

[0141] (12)

[0142] S224, the simultaneous equations (11) (12) are obtained

[0143] (13)

[0144] (14)

[0145] S225, let , , formula (14) can be simplified to , the unknown is the relationship matrix between the hand and the eye, and the classic Tsai two-step method can be used to solve matrix equation.

[0146] S226, the conversion matrix of the image coordinate system to the camera coordinate system can be obtained through the camera external parameter . The image coordinates of the target object are converted to the depth camera coordinate system, and the calculation formula (15) is as follows:

[0147] (15)

[0148] S227, given the hand-eye relationship matrix and the conversion matrix of the end effector coordinate system to the mechanical arm base coordinate system read from the mechanical arm system , the conversion matrix of the camera coordinate system to the mechanical arm base coordinate system can be obtained, and formula (16) is used to convert the coordinates of the target object to the mechanical arm base coordinate system to obtain the spatial coordinates of the target object in the mechanical arm base coordinate system.

[0149] (16)

[0150] S230: Obtain the original mechanical arm position information, control the mechanical arm to move to a specified first spatial position, and perform picking work on the target object. Specifically, the mechanical arm randomly selects a pickable target fruit with a shielding form of unshielded (NO) or leaf shielding (OL) to perform the picking task; according to the identification result and positioning result of the target object in multiple groups of picking environment images, combined with the corresponding mechanical arm position information, the next motion starting point of the mechanical arm is calculated.

[0151] S231: When performing the picking task, control the depth camera to acquire a plurality of sets of multi-modal images of the target on the motion path, and record the position information of the robot arm acquiring each set of multi-modal images of the target.

[0152] Specifically, while performing the selective picking task, the depth camera acquires a plurality of sets of multi-modal images of the target fruit trees on the motion path at a rate of 15 FPS, and records the position information of the robot arm acquiring each set of multi-modal images of the target fruit trees.

[0153] S232, a plurality of sets of multi-modal images of the target fruit trees on the motion path of the robot arm V is represented by formula (17), v 1 to v n representing the first set to the nth set of multi-modal images of the target fruit trees:

[0154] (17)

[0155] S233, sequentially inputting a plurality of sets of multi-modal images of the target into the target detection model, outputting a plurality of classification labels and boundary box coordinates of the target; setting the unobstructed and non-influencing object as a pickable object (PO); setting the object obstructed by the influencing object as a non-pickable object (NPO). Subsequently, the number of pickable objects (PO) and non-pickable objects (NPO) in each set of multi-modal images is calculated , and the pickable rate of the target fruit in each set of multi-modal images is calculated by formula (18) , v j representing the first V th in the set of multi-modal images of the target fruit trees j

[0156] (18)

[0157] S234, calculating the set of multi-modal images with the highest pickable rate of the target fruit , and the calculation formula is formula (19):

[0158] (19)

[0159] S235, taking the robot arm position information corresponding to the set of multi-modal images as the next motion starting point of the robot arm, and controlling the robot arm to move to the motion starting point when performing the next picking task.

[0160] S236: communicating with the robot arm to acquire the position information of the robot arm, and controlling the robot arm to move to a specified first spatial position, the robot arm being used for picking work on the target. ​

[0161] S240: repeat S210 to S230 until there is no pickable object in the field of view of the depth camera carried by the mechanical arm, and the picking operation is completed.

[0162] Exemplary system

[0163] As shown in Figure 8 , a multi-modal data fusion-based in-body fruit picking system, in a specific application process, the depth camera is installed and fixed at the end of the mechanical arm, the depth camera and the mechanical arm module are in communication connection with the main controller, and the main controller comprises the following:

[0164] An image processing module 20 is used to obtain multi-modal visual data shot by the depth camera, and the multi-modal visual data contains a target object to be picked; a target detection model trained by using the above-mentioned multi-modal data fusion-based target detection method is used to identify and locate the target object in the multi-modal visual data, and output a plurality of classification labels and boundary box coordinates of the target object, the classification labels including three types of unobstructed, obstructed by non-influencing objects and obstructed by influencing objects.

[0165] A spatial coordinate conversion module 30 is used to obtain first spatial position coordinates of the boundary box coordinates of the target object relative to the depth camera; and convert the spatial position coordinates of the target object relative to the depth camera into mechanical arm base position coordinate information.

[0166] A motion control module 40 is used to obtain the original mechanical arm position information, control the mechanical arm to move to a specified first spatial position, and perform picking operation on the target object; randomly select one classification label as unobstructed or obstructed by non-influencing objects as a pickable target fruit to execute the picking task.

[0167] A motion calculation module 50 is used to calculate the next motion starting point of the mechanical arm according to the identification result and the positioning result of the target object in a plurality of groups of picking environment images, in combination with the corresponding mechanical arm position information.

[0168] Exemplary electronic device

[0169] Next, the electronic device according to the embodiments of the present application will be described with reference to Figure 4 . The electronic device can be a mobile device itself, or a single device independent of the mobile device, which can communicate with the mobile device to receive the collected input signals therefrom and send the selected target decision behaviors thereto.

[0170] Figure 4 A block diagram of the electronic device according to the embodiments of the present application is shown.

[0171] As shown in Figure 4As shown, the electronic device 10 includes one or more processors 11 and a memory 12.

[0172] The processor 11 can be a central processing unit (CPU) or other form of processing unit that has data processing and / or instruction execution capabilities, and can control other components in the electronic device 10 to perform desired functions.

[0173] The memory 12 can include one or more computer program products that can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory, for example, can include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage media, and the processor 11 can run the program instructions to implement the decision-making behavior decision-making method of various embodiments of the present application described above and / or other desired functions.

[0174] In one example, the electronic device 10 can further include an input device 13 and an output device 14, which are interconnected through a bus system and / or other form of connection mechanism (not shown). The input device 13, for example, can include various devices such as an on-board diagnostic system (OBD), a unified diagnostic service (UDS), an inertial measurement unit (IMU), a camera, a laser radar, a millimeter wave radar, an ultrasonic radar, a vehicle-to-everything (V2X) communication, and / or the like. The input device 13 can also include, for example, a keyboard, a mouse, and / or the like. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and / or the like.

[0175] Of course, in order to simplify, Figure 4 Only some of the components in the electronic device 10 related to the present application are shown in the figure, and components such as buses, input / output interfaces, and / or the like are omitted. In addition, the electronic device 10 can include any other appropriate components according to specific application cases.

[0176] Exemplary computer program product and computer-readable storage medium

[0177] In addition to the above-mentioned methods and devices, embodiments of the present application can also be computer program products that include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the decision-making behavior decision-making method according to various embodiments of the present application described in the “Exemplary Method” section of the present specification.

[0178] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. The embodiments of the present application are not limited by the

[0179] Furthermore, embodiments of the present application can also be a computer readable storage medium, having stored thereon computer program instructions which, when executed by a processor, cause the processor to carry out the steps described in the above "Exemplary Method" section of the present specification for the decision-making method according to various embodiments of the present application.

[0180] The computer readable storage medium can be any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0181] The above describes the basic principles of the present application in combination with specific embodiments, but it should be noted that the advantages, benefits, effects and the like mentioned in the present application are only examples and are not limiting, and these advantages, benefits, effects and the like cannot be considered as necessary for each embodiment of the present application. In addition, the above specific details are only for the purpose of example and understanding, and the above details do not limit the present application to the above specific details.

[0182] The block diagrams of the devices, apparatuses, equipment, systems referred to in this application are only illustrative examples and are not intended to require or imply that the connection, arrangement, configuration must be as shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "comprise", "have", etc. are open-ended words that are to be interpreted in the context where they are used. They are not meant to be limiting. The word "or" as used in this document is intended to mean "and / or" unless otherwise indicated. The word "and" as used in this document is intended to mean "and / or" unless otherwise indicated. The word "such as" is used in this document to mean "such as but not limited to" and is intended to mean that what follows is an example of what is included in the term being used.

[0183] It is also important to note that the devices, apparatuses and methods of the present application can be embodied in a variety of other forms, including but not limited to a device, apparatus, system, method, process, computer-readable medium, computer program product, and the like. It is also important to note that the devices, apparatuses and methods of the present application can be embodied as one or more components, either alone or in combination with other components, including but not limited to a device, apparatus, system, method, process, computer-readable medium, computer program product, and the like.

[0184] The above description of disclosed aspects is given for illustrative purposes and is not intended to limit the scope of the application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0185] The above description has been given for illustrative and descriptive purposes. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although several example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A target detection method based on multi-modal data fusion, characterized in that, The method comprises the following steps: S110: constructing a target detection model based on multi-modal data fusion, the target detection model comprising a feature extraction network, a neck structure and a prediction layer, a coordinate attention mechanism being introduced into the feature extraction network and the neck structure to enhance the perception ability of the target object, wherein the multi-modal visual data comprises multi-modal visual data of RGB images, infrared images and depth images; S120: obtaining multi-modal visual data of the in-vivo target object as a training set, and inputting the target detection model for training to obtain a trained target detection model; S130: obtaining to-be-detected multi-modal visual data, the to-be-detected multi-modal visual data comprising a target object to be picked; The to-be-detected multi-modal visual data is input into the target detection model to identify the target object in the multi-modal visual data, and an identification result is output; the target detection model adopts a YOLOv5s model, the feature extraction network of the YOLOv5s model adopts a depth separable convolution to replace an ordinary convolution in a CBL module to form a DPBL module, and a coordinate attention module is embedded after a CSP_1_X module cascade operation; The neck structure comprises a DPBL module and a CSP_2_X module, the CSP_2_X module being formed by cascading an ordinary convolution and X Res units, and a coordinate attention module being embedded at the end of the neck structure; The channel number of the Focus module structure of the feature extraction network is adjusted to five channels to realize reading of the multi-modal visual data. 2.The target detection method based on multi-modal data fusion according to claim 1, characterized in that, The coordinate attention mechanism is introduced into the feature extraction network and the neck structure to enhance the perception ability of the target object, which comprises embedding a coordinate attention module in the feature extraction network and embedding a coordinate attention module at the end of the neck structure. 3.The target detection method based on multi-modal data fusion according to claim 2, characterized in that, The step S120 comprises: configuring target detection model training key parameters, the key parameters comprising the size of an input image, the number of categories, the number of training iterations, the initial learning rate, the learning rate adjustment strategy and the number of input images per batch processing; Adopting CIOU Loss As a bounding box loss function, the target detection model is trained to converge, the weight file obtained after each training iteration is saved at the end of training, the weight file with the best validation effect is saved, and thus a trained target detection model is obtained.

4. A method for in vivo fruit picking based on a target detection model, characterized by, The method comprises the following steps: S210: obtaining an identification result according to the target detection method based on multi-modal data fusion according to any one of claims 1-3, the identification result comprising a plurality of classification labels and a bounding box coordinate of a target object, the classification labels comprising three types of unobstructed, obstructed by a non-influencing object and obstructed by an influencing object; S220: randomly selecting a target object with an unobstructed or non-influencing object obstructed form as a picking object, obtaining a first spatial position coordinate of the bounding box coordinate of the picking object relative to a depth camera; converting the spatial position coordinate of the picking object relative to the depth camera into mechanical arm base position coordinate information; S230: obtaining original mechanical arm position information, controlling the mechanical arm to move to a specified first spatial position for picking work on the picking object; According to the identification result and the positioning result of the target object in a plurality of groups of picking environment images, the corresponding mechanical arm position information is combined to calculate a next motion starting point of the mechanical arm; S240: repeating the above S210 to S230 until there is no pickable object in the field of view of the depth camera carried by the mechanical arm, and the picking work is completed.

5. The in-body fruit picking method based on a target detection model according to claim 4, characterized in that, The step S210 further comprises: receiving multi-modal visual data of picking environment images captured in real time by the depth camera, the multi-modal visual data comprising RGB images, depth images and infrared images; performing channel fusion on the RGB images, the depth images and the infrared images, and setting the image resizing to a predetermined size to obtain multi-modal images; inputting the multi-modal images into the target detection model to output the boundary box coordinates and classification labels corresponding to the target objects and the corresponding confidence; according to the confidence threshold judgment, removing the prediction results with confidence less than the confidence threshold to obtain the boundary box coordinates and class probability of the target fruits; using a non-maximum suppression algorithm to remove redundant boundary boxes on the same target object to obtain the prediction results.

6. The in-body fruit picking method based on a target detection model according to claim 5, wherein The step S230 further comprises: During the picking operation, controlling the depth camera to obtain multi-modal images of multiple targets on the motion path and recording the mechanical arm position information of each group of multi-modal images of the targets; The multi-modal image set V of multiple targets on the motion path of the mechanical arm is represented as follows: wherein, to representing the multi-modal images of the target fruit trees from the first group to the nth group; inputting the multi-modal images of the target in multiple groups into the target detection model in turn, outputting a plurality of classification labels and boundary box coordinates corresponding to the target; setting the unobstructed and the obstructed by the non-influencing object as the pickable object (PO); setting the obstructed by the influencing object as the unpickable object (NPO); The number of pickable objects (PO) and non-pickable objects (NPO) in each set of multi-modal images is calculated, and the pickability of target fruits in each set of multi-modal images is calculated by the following formula wherein, represents the jth in the target fruit tree multi-modal image set V; a group of multi-modal images with the highest pickability of the target fruit is calculated out , and the calculation formula is as follows: The mechanical arm position information corresponding to the group of multi-modal images is used as the next motion starting point of the mechanical arm, and the mechanical arm is controlled to move to the motion starting point when performing the next picking task.

7. A target detection model-based in-body fruit picking system, characterized by, It comprises: An image processing module for obtaining an identification result according to the target detection method based on multi-modal data fusion according to any one of claims 1-3, the identification result comprising multiple classification labels and boundary box coordinates of target objects, and the classification labels comprising three types of unobstructed, obstructed by non-influencing objects and obstructed by influencing objects; A spatial coordinate conversion module for obtaining first spatial position coordinates of the boundary box coordinates of the target objects relative to the depth camera, and converting the spatial position coordinates of the target objects relative to the depth camera into mechanical arm base position coordinate information; A motion control module for obtaining original mechanical arm position information, controlling the mechanical arm to move to a specified first spatial position for picking operation on the target objects, and randomly selecting a classification label as unobstructed or obstructed by non-influencing objects as a pickable target fruit to perform a picking task; A motion calculation module for calculating the next motion starting point of the mechanical arm according to the identification result and positioning result of the target objects in multiple groups of picking environment images combined with the corresponding mechanical arm position information.

8. An electronic device, comprising: It comprises a processor, an input device, an output device and a memory, which are connected in sequence, the memory is used to store a computer program, the computer program comprises program instructions, the processor is configured to invoke the program instructions to execute the method according to any one of claims 1-6.

9. A readable storage medium, characterized by, The storage medium stores a computer program, the computer program comprises program instructions, and the program instructions make the processor execute the method according to any one of claims 1-6 when the processor executes the program instructions.

Images