Nondestructive detection method and system for litchi quality based on multimodal fusion

Abstract

This invention discloses a method and system for nondestructive testing of litchi quality based on multimodal fusion. The method comprises: acquiring multiple sets of litchi RGB images and litchi depth images from different angles to construct a litchi dataset; establishing an improved CenterNet network model and employing a multimodal data fusion strategy; performing distributed training on the improved CenterNet network model based on the litchi dataset and optimizing the training process; and inputting the captured image stream into the trained improved CenterNet model to implement nondestructive testing of litchi quality. This method addresses the inefficiency, lack of real-time performance, and insufficient accuracy of traditional fruit quality testing methods. Furthermore, the present invention focuses on key local areas of the image, rapidly capturing critical information within the image, providing theoretical support for the research of online nondestructive testing technology and equipment for litchi quality.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and smart agriculture technology, and more specifically to a non-destructive detection method and system for litchi quality based on multimodal fusion. Background Art

[0002] As a key cash crop in southern China, lychees play a significant role in boosting the local economy and farmers' income. Lychees typically mature between May and August, necessitating real-time monitoring of their quality to ensure optimal harvesting.

[0003] Traditional lychee orchard fruit quality monitoring relies primarily on subjective judgment by fruit growers, which is not only time-consuming and inefficient, but also leads to time lags and complexities in subsequent lychee yield estimation and harvesting. With the rapid development of computer vision and smart agriculture, there is an urgent need to develop a non-destructive lychee quality testing method that can help fruit growers automatically monitor lychee quality, promote the intelligent development of the lychee industry, and improve farmers' economic benefits and quality of life. Summary of the Invention

[0004] In view of this, the present invention provides a non-destructive detection method and system for litchi quality based on multimodal fusion, which solves the problems of low efficiency, lack of real-time performance and insufficient accuracy of traditional fruit quality detection methods. The present invention also focuses on local key areas of the image and quickly captures key information in the image, providing theoretical support for the research of online non-destructive detection technology and equipment for litchi quality.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A nondestructive detection method for litchi quality based on multimodal fusion, comprising:

[0007] Obtain multiple sets of litchi RGB images and litchi depth images from different angles to construct a litchi dataset;

[0008] Establish an improved CenterNet network model and adopt a multimodal data fusion strategy;

[0009] Conduct distributed training on the fused improved CenterNet network model based on the Lizhi dataset and optimize the training process;

[0010] The captured image stream is input into the trained improved CenterNet model to achieve non-destructive testing of litchi quality.

[0011] Preferably, an improved CenterNet network model is established, and a multimodal data fusion strategy is adopted including:

[0012] CenterNet is used as the basic framework of the litchi quality non-destructive testing model;

[0013] The lightweight network ConvNeXt-tiny is used as the pure convolutional architecture backbone network of the litchi quality non-destructive testing model;

[0014] Add a CFP centralized pyramid module at the end of the pure convolutional architecture backbone network;

[0015] The litchi RGB image and litchi depth image at the same angle are registered with the same pixel width, and the single-channel litchi depth image is encoded into a three-channel HHA image;

[0016] The registered RGB image and HHA image are used as three-channel inputs, and feature extraction is performed separately through a ConvNeXt-tiny pure convolutional architecture backbone network. The two features are fused in the middle layer and input into the CFP centralized pyramid module in a parallel nested manner to achieve multimodal data fusion.

[0017] Preferably, the lightweight network ConvNeXt-tiny feature extraction process is:

[0018] The lightweight network ConvNeXt-tiny first preprocesses the input image, including a layer normalization LN and a convolution of size 4 and stride 4, with an output channel number of 96;

[0019] It passes through 4 stages with channel numbers of (96, 192, 384, 768) and corresponding stacking ratios of [3, 3, 9, 3]. The first stage includes the ConvNeXt Block1 module, the second stage includes the downsampling layer and the ConvNeXt Block2 module, the third stage includes the downsampling layer and the ConvNeXt Block3 module, and the fourth stage includes the downsampling layer and the ConvNeXt Block4 module. The ConvNeXt Block1 module, the ConvNeXt Block2 module, the ConvNeXt Block3 module and the ConvNeXtBlock4 module all include a 7×7 depthwise separable convolution layer, two multi-layer perceptrons (MLPs) and two 1×1 convolution layers. The Gaussian error linear unit activation function (GELU) is used in the two 1×1 convolution layers. The downsampling layer includes an LN layer and a convolution layer with a convolution kernel size of 2 and a stride of 2.

[0020] Preferably, performing image registration on the litchi RGB image and the litchi depth image specifically includes:

[0021] The pixels of the litchi RGB image are represented as (u R ,v R ,z R ) T, the pixel of the litchi depth image is represented as (u L ,v L ,z L ) T , where u, v, and z represent the horizontal and vertical coordinates of the image and the depth value in the camera coordinate system, respectively, and T represents transposition. The registration formula is as follows:

[0022]

[0023] W is a 4×4 matrix expressed as:

[0024]

[0025] Among them, RR is the intrinsic parameter matrix of the high-definition camera calibrated by the binocular camera, LR is the intrinsic parameter matrix of the depth camera calibrated by the binocular camera, M is the extrinsic parameter matrix of the optical centers of the two cameras, and r11~r44 are the calculated parameters.

[0026] Expand the registration formula to register the depth pixel points to the corresponding positions of the RGB pixel points, which can be expressed as:

[0027]

[0028]

[0029] z R ≈z L

[0030] Preferably, multimodal data fusion specifically includes:

[0031] The size of the registered RGB image and HHA image is 640×640, and they are respectively input into the ConvNeXt-tiny pure convolutional architecture backbone network for feature extraction. The ConvNeXt-tiny pure convolutional architecture backbone network contains four stages of conv, and the sizes of the feature maps obtained in the four different stages are 160×160, 80×80, 40×40, and 20×20 respectively;

[0032] Construct an intermediate layer feature fusion network to fuse feature information. The intermediate layer fusion network uses add and 1×1 convolution kernels to fuse feature information.

[0033] In addition to the first stage, the RGB image depth feature map and the HHA image depth feature map obtained by the second stage conv are passed through the intermediate layer feature fusion network and input into layer3 of the CFP centralized pyramid module;

[0034] The RGB image depth feature map obtained by the third stage conv and the HHA image depth feature map are passed through the intermediate layer feature fusion network and input into layer4 of the CFP centralized pyramid module;

[0035] The RGB image depth feature map and the HHA image depth feature map obtained in the fourth stage conv are passed through the intermediate layer feature fusion network, and then downsampled by 2 times and input into layer 1 of the CFP centralized pyramid module;

[0036] The CFP centralized pyramid module fuses the input feature maps of different stages, combines them with the feature data passed in by the previous unit to perform operations, and obtains the final feature representation, thus realizing multimodal data feature fusion.

[0037] Preferably, the optimization training process specifically includes:

[0038] Input the registered RGB image and HHA map into the improved CenterNet network model for synchronous training;

[0039] Use SGD+Momentum optimizer for target optimization;

[0040] The DDP mode is adopted and gradient clipping is introduced to prevent gradient explosion.

[0041] Preferably, constructing the litchi dataset specifically includes:

[0042] The vehicle uses a high-definition camera and a depth camera to form a binocular camera, which captures litchi trees in the orchard from multiple angles. Each litchi tree has multiple sets of RGB images and depth images from different angles.

[0043] By manually screening and labeling litchi RGB images and litchi depth images, litchi without any damage was labeled as healthy litchi, and litchi with obvious damage on the outer surface of the peel was labeled as damaged litchi;

[0044] Supervised data augmentation is performed on all annotated images. Specific data augmentation methods include random rotation, random flipping, color jittering, color gamut transformation, and noise superposition.

[0045] Preferably, encoding the single-channel litchi depth image into a three-channel HHA image specifically includes:

[0046] The single-channel Lizhi depth image is encoded into three channels through HHA encoding: horizontal disparity, height above the ground, and norm angle;

[0047] The three-channel data of horizontal disparity, height above the ground and norm angle are linearly scaled to 0-255 to obtain a three-channel HHA image.

[0048] A nondestructive detection system for litchi quality based on multimodal fusion, comprising: a data acquisition module, a model building module and a detection module;

[0049] Data acquisition module: used to obtain multiple sets of litchi RGB images and litchi depth images from different angles to construct a litchi dataset;

[0050] Model building module: used to build an improved CenterNet network model, using a multimodal data fusion strategy;

[0051] Detection module: used to perform distributed training on the improved CenterNet network model based on the Lizhi dataset and optimize the training process;

[0052] The captured image stream is input into the trained improved CenterNet model to achieve non-destructive testing of litchi quality.

[0053] A terminal device, comprising:

[0054] memory for storing computer programs; and

[0055] A processor, configured to implement the litchi quality nondestructive testing method according to any one of claims 1 to 8 when executing the computer program.

[0056] The present invention discloses a method and system for nondestructive detection of litchi quality based on multimodal fusion. Compared with the prior art, the present invention has the following advantages:

[0057] 1. Higher recognition accuracy: The system uses a depthwise separable convolutional module and a CFP feature pyramid structure, along with multimodal data input, to extract more accurate features, resulting in higher recognition accuracy in litchi damage detection tasks.

[0058] 2. Higher detection efficiency: The lightweight ConvNeXt-tiny backbone network structure is adopted, and the model training process is optimized. This can achieve higher detection efficiency and faster recognition speed while ensuring recognition accuracy.

[0059] 3. Better adaptability and generalization: This approach uses the grouped convolution concept of ConvNeXt and the multi-scale feature fusion of CFP, enabling it to adapt to litchi images of different scales, poses, and lighting conditions, resulting in better adaptability and generalization.

[0060] In summary, the present invention achieves the purpose of nondestructive detection of litchi quality through technical means such as multimodal data fusion strategy, improved CenterNet network model, automation and nondestructive testing, image processing technology, distributed training and optimization. It has strong practicality and promotion value and can provide a better solution for litchi identification tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0062] Figure 1 The accompanying figure is a flow chart of a nondestructive detection method for litchi quality based on multimodal fusion provided by the present invention.

[0063] Figure 2 The accompanying drawing is a schematic diagram of the structure of the multimodal data fusion model provided by the present invention.

[0064] Figure 3 The accompanying figure is a schematic diagram of the overall structure of the improved CenterNet model provided by the present invention.

[0065] Figure 4 The accompanying figure is a schematic diagram of the Block module of the backbone network ConvNeXt-tiny provided by the present invention, wherein Figure 4 (a) shows the specific structure of ConvNeXt Block1 module. Figure 4 (b) shows the specific structure diagram of ConvNeXt Block2 module. Figure 4 (c) shows the specific structure diagram of ConvNeXt Block3 module. Figure 4 (d) shows the specific structure diagram of the ConvNeXt Block4 module.

[0066] Figure 5 The accompanying drawing is a schematic diagram of the network structure of the backbone network ConvNeXt-tiny provided by the present invention.

[0067] Figure 6 The accompanying figure is a schematic diagram of the structure of the litchi quality nondestructive detection system based on multimodal fusion provided by the present invention. DETAILED DESCRIPTION

[0068] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0069] The embodiment of the present invention discloses a nondestructive detection method for litchi quality based on multimodal fusion, such as Figure 1 As shown, including:

[0070] Obtain multiple sets of litchi RGB images and litchi depth images from different angles to construct a litchi dataset;

[0071] Establish an improved CenterNet network model and adopt a multimodal data fusion strategy;

[0072] Conduct distributed training on the improved CenterNet network model based on the Lizhi dataset and optimize the training process;

[0073] The captured image stream is input into the trained improved CenterNet model to achieve non-destructive testing of litchi quality.

[0074] In this embodiment, the specific process of collecting images and constructing data sets is as follows:

[0075] The mountain orchard cart is equipped with a high-definition camera and a depth camera. The high-definition camera and the depth camera form a binocular camera to take multi-angle photos of the lychee trees in the lychee orchard. The mountain orchard cart is equipped with multiple sensors, and local path planning is carried out based on the lidar, and multiple directions of photography are taken around the lychee trees. The light intensity is adjusted based on the photoelectric sensor to reduce light interference in complex environments. After screening, each lychee tree has multiple sets of RGB images and depth images at different angles. According to actual needs, this embodiment uses 5 sets of RGB images and depth images at different angles. The high-definition camera obtains the RGB image, and the depth camera obtains the depth image.

[0076] During the image screening stage, the images were manually screened to eliminate images with low resolution, severe fruit occlusion, or shooting distance, and then re-photographed at the same angle as the discarded images. The LabelImg annotation tool was used for manual data annotation, and lychees without any damage were labeled as healthy lychees, and lychees with obvious damage on the outer surface of the peel were labeled as damaged lychees. The annotation file was stored as an xml file and then converted into a json file in the CenterNet training format. The final database contained lychee images with few fruits, many fruits, sunny days, and cloudy days, providing a rich and diverse data source for subsequent training and testing. The lychee RGB images and Depth images were divided into training set, validation set, and test set in a ratio of 8:1:1;

[0077] Supervised data augmentation is used on all litchi images to increase the sample size of the litchi dataset and improve the robustness and generalization of the network model. Specific data augmentation methods include random rotation, random flipping, color jittering, color gamut transformation, and noise superposition.

[0078] In this embodiment, an improved CenterNet network model is established, such as Figure 2 Shown, including:

[0079] The anchor-free target detection algorithm CenterNet is used as the basic framework of the litchi fruit detection model;

[0080] The lightweight network ConvNeXt-tiny is used as the pure convolutional architecture backbone network of the litchi quality non-destructive testing model;

[0081] Add a CFP centralized pyramid module at the end of the pure convolutional architecture backbone network;

[0082] The litchi RGB image and litchi depth image at the same angle are registered with the same pixel width to ensure a one-to-one relationship between the pixels, and the single-channel litchi depth image is encoded into a three-channel HHA image.

[0083] The registered RGB image and HHA image are used as three-channel inputs, and feature extraction is performed separately through a ConvNeXt-tiny pure convolutional architecture backbone network. The two features are fused in the middle layer and input into the CFP centralized pyramid module in a parallel nested manner to achieve multimodal data fusion.

[0084] In this embodiment, the original backbone feature extraction network ResNet-18 of CenterNet is replaced with ConvNeXt-tiny, as shown in Figure 3 As shown in the figure, the feature extraction process of the lightweight network ConvNeXt-tiny is:

[0085] The lightweight network ConvNeXt-tiny first preprocesses the input image, including a layer normalization (LN) and a convolution with a size of 4 and a stride of 4, resulting in 96 output channels. It then passes through four stages with channel numbers of (96, 192, 384, 768) and stacking ratios of [3, 3, 9, 3]. The first stage consists of a single ConvNeXt Block module, while each of the remaining stages consists of a downsampling layer and a ConvNeXt Block module. Specifically, the first stage includes ConvNeXt Block1, the second stage includes a downsampling layer and ConvNeXt Block2, the third stage includes a downsampling layer and ConvNeXt Block3, and the fourth stage includes a downsampling layer and ConvNeXt Block4.

[0086] like Figure 4 As shown in the figure, each ConvNeXt Block module includes a 7×7 depth-separable convolution layer, two multi-layer perceptrons MLP and two 1×1 convolution layers, that is, the ConvNeXt Block1 module, ConvNeXt Block2 module, ConvNeXtBlock3 module and ConvNeXt Block4 module all include a 7×7 depth-separable convolution layer, two multi-layer perceptrons MLP and two 1×1 convolution layers. The Gaussian error linear unit activation function GELU is used in the two 1×1 convolution layers. The downsampling layer includes an LN layer and a convolution layer with a convolution kernel size of 2 and a stride of 2. After the fourth residual structure stacking, the feature extraction of the network is completed.

[0087] In this embodiment, a CFP centralized pyramid module is added to the end of a purely convolutional backbone network. The CFP centralized pyramid module consists of multiple submodules, each responsible for acquiring features at a specific scale, then aggregating these features at a central location for processing, and ultimately outputting a unified feature map. Each feature pyramid layer includes multiple feature layers. High-level feature layers have a larger receptive field and lower resolution, enabling detection of large-scale objects; low-level feature layers have a smaller receptive field and higher resolution, enabling detection of small-scale objects. The CFP centralized pyramid module primarily consists of an input layer, a backbone for extracting a visual feature pyramid, a proposed explicit visual center (EVC), a proposed global centralized regulation (GCR), and a decoupled head network for object detection (including classification, regression, and segmentation losses). EVC and GCR are implemented on the extracted feature pyramid. First, the input image is fed into the backbone network to extract a five-layer feature pyramid X, where the spatial size of each layer of features Xi (i = 0, 1, 2, 3, 4) is 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image, respectively.

[0088] Specifically, such as Figure 5 As shown in Figure 2, multimodal data fusion specifically includes:

[0089] The size of the registered RGB image and HHA image is 640×640, and they are respectively input into the ConvNeXt-tiny pure convolutional architecture backbone network for feature extraction. The number of input image channels is 3. The ConvNeXt-tiny pure convolutional architecture backbone network contains 4 stages of conv. The sizes of the feature maps obtained in the 4 different stages are 160×160, 80×80, 40×40, and 20×20 respectively.

[0090] Construct an intermediate layer feature fusion network to fuse feature information. The intermediate layer fusion network uses add and 1×1 convolution kernels to fuse feature information.

[0091] In addition to the first stage, the RGB image depth feature map and the HHA image depth feature map obtained by the second stage conv are passed through the intermediate layer feature fusion network and input into layer3 of the CFP centralized pyramid module;

[0092] The RGB image depth feature map obtained by the third stage conv and the HHA image depth feature map are passed through the intermediate layer feature fusion network and input into layer4 of the CFP centralized pyramid module;

[0093] The RGB image depth feature map and the HHA image depth feature map obtained in the fourth stage conv are passed through the intermediate layer feature fusion network, and then downsampled by 2 times and input into layer 1 of the CFP centralized pyramid module;

[0094] The CFP centralized pyramid module fuses the input feature maps of different stages, combines them with the feature data passed in by the previous unit to perform operations, and obtains the final feature representation, thus realizing multimodal data feature fusion.

[0095] In this embodiment, the add method is used to fuse features, which can be expressed as:

[0096] Assuming that the channels of the two inputs are X1 and Y1, the single output channel of concat is:

[0097] Z add =(X1+Y1)*K1

[0098] Here, * represents the convolution operation, and K1 represents the convolution kernel. When using the add method to perform feature fusion, the dimension and shape of the features are not changed, thus preserving the original feature information.

[0099] In this embodiment, the litchi RGB image and the litchi depth image are registered, specifically including:

[0100] The pixels of the litchi RGB image are represented as (u R ,v R ,z R ) T , the pixel of the litchi depth image is represented as (u L ,v L ,z L ) T , where u, v, and z represent the horizontal coordinate, vertical coordinate, and depth value of the camera coordinate system, respectively. The registration formula is as follows:

[0101]

[0102] W is a 4×4 matrix expressed as:

[0103]

[0104] Among them, RR is the intrinsic parameter matrix of the right high-definition camera calibrated by the binocular camera, LR is the intrinsic parameter matrix of the left depth camera calibrated by the binocular camera, M is the extrinsic parameter matrix of the optical centers of the two cameras, and r11~r44 are the calculated parameters.

[0105] Expand the registration formula to register the depth pixel points to the corresponding positions of the RGB pixel points, which can be expressed as:

[0106]

[0107]

[0108] z R ≈z L

[0109] In this embodiment, encoding the single-channel litchi depth image into a three-channel HHA image specifically includes:

[0110] The single-channel Lizhi depth image is encoded into three channels through HHA encoding: horizontal disparity, height above the ground, and norm angle;

[0111] The three-channel data of horizontal disparity, height above the ground and norm angle are linearly scaled to 0-255 to obtain a three-channel HHA image.

[0112] In this embodiment, the optimization training process specifically includes:

[0113] Input the registered RGB image and HHA map into the improved CenterNet network model for synchronous training;

[0114] Use SGD+Momentum optimizer for target optimization;

[0115] The DDP mode is adopted and gradient clipping is introduced to prevent gradient explosion.

[0116] The DDP mode is used in distributed training and gradient clipping is introduced, including:

[0117] The DDP mode increases parallelism by increasing the batch size, improves communication efficiency through the Ring-Reduce data exchange method, and alleviates the limitations of the Python GIL by starting multiple processes, thereby increasing training speed.

[0118] In order to avoid gradient explosion, the present invention introduces gradient clipping during the training process. If ||g||≥c, it can be adjusted as follows:

[0119] g←c·g / ||g||

[0120] Where c is a hyperparameter, g is the gradient, ||g|| is the norm of the gradient, and g / ||g|| must be a unit vector.

[0121] This embodiment discloses a nondestructive detection system for litchi quality based on multimodal fusion, such as Figure 6 As shown, it includes: data acquisition module, model building module and detection module;

[0122] Data acquisition module: used to obtain multiple sets of litchi RGB images and litchi depth images from different angles to construct a litchi dataset;

[0123] Model building module: used to build an improved CenterNet network model, using a multimodal data fusion strategy;

[0124] Detection module: used to perform distributed training on the improved CenterNet network model based on the Lizhi dataset and optimize the training process;

[0125] The captured image stream is input into the trained improved CenterNet model to obtain detection results and count the number of damaged fruits in the area; the health status of the fruit trees is monitored and early warning is carried out based on the proportion of damaged fruits in the image.

[0126] In this embodiment, the specific process of collecting multiple groups of litchi RGB images and litchi depth images at different angles is as follows: the on-board high-definition camera and the depth camera are fixedly installed on the mountain orchard vehicle body and electrically connected to the mountain orchard vehicle body; the mountain orchard vehicle body is communicatively connected to the terminal device, and the mountain orchard vehicle body is equipped with the on-board high-definition camera and drives to the designated location area, and performs the automatic driving task according to the automatic driving route planned by the sensor; the on-board depth camera is used to capture depth image data, and the on-board high-definition camera is used to capture litchi RGB images; the integrated spectrum LED light source is used to automatically supplement and adjust the light source according to the feedback from the photoelectric sensor when shooting images, and the depth image and RGB image data are transmitted to the terminal device through the mountain orchard vehicle body.

[0127] In this embodiment, the driving photography task needs to be performed in a sunny, cloudless or lightly cloudy environment with no wind or low wind speed.

[0128] The high-definition camera used in this embodiment can be an HQ7620 camera module based on OV7620. OV7620 is a color CMOS camera chip from Omnivision with a resolution of up to 664×492 and a transmission rate of up to 30 frames per second (f / s).

[0129] The depth camera used in this embodiment can be the Argus100L structured light depth camera produced by Xi'an Zhiwei Technology Co., Ltd., which uses MEMS dynamic infrared stripe structured light projection, obtains grayscale information through CMOS, and calculates depth information through changes in projection pattern.

[0130] The integrated spectrum LED light source used in this embodiment is set to have a continuous high-intensity distribution of light intensity with frequency changes within the 550-1000 nm band, and can emit directional light with a total power of only 72 mW.

[0131] The orchard mountain bike used in this example is built based on the XAG R1502022 agricultural unmanned vehicle. It adopts four-wheel drive, has a rated working speed of 1 meter per second, a minimum turning radius of 1 meter, and a maximum climbing angle of 15.

[0132] Compared with traditional non-destructive lychee quality testing technologies, this method requires less manpower and resources. Once the model is built, it only requires regular driving and filming using a mountain orchard vehicle equipped with an onboard high-definition camera and a depth camera. This allows for efficient, accurate, and rapid testing, making it suitable for the management and maintenance of lychee orchards and providing farmers with more scientific planting and management plans. In addition to lychees, this method can also be applied to the non-destructive quality monitoring of other crops, such as dragon fruit, apples, kiwis, and tomatoes, demonstrating its broad potential for application.

[0133] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0134] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A nondestructive detection method for litchi quality based on multimodal fusion, characterized in that: include: Obtain multiple sets of litchi RGB images and litchi depth images from different angles to construct a litchi dataset; Establish an improved CenterNet network model and adopt a multimodal data fusion strategy; Conduct distributed training on the fused improved CenterNet network model based on the Lizhi dataset and optimize the training process; The captured image stream is input into the trained improved CenterNet model to achieve non-destructive testing of litchi quality; An improved CenterNet network model is established, and the multimodal data fusion strategy includes: CenterNet is used as the basic framework of the litchi quality non-destructive testing model; The lightweight network ConvNeXt-tiny is used as the pure convolutional architecture backbone network of the litchi quality non-destructive testing model; Add a CFP centralized pyramid module at the end of the pure convolutional architecture backbone network; The litchi RGB image and litchi depth image at the same angle are registered with the same pixel width, and the single-channel litchi depth image is encoded into a three-channel HHA image; The registered RGB image and HHA image are used as three-channel inputs, and feature extraction is performed on each of them through a ConvNeXt-tiny pure convolutional architecture backbone network. The two features are fused in the middle layer and input into the CFP centralized pyramid module in a parallel nested manner to achieve multimodal data fusion. Multimodal data fusion specifically includes: The size of the registered RGB image and HHA image is 640×640, and they are respectively input into the ConvNeXt-tiny pure convolutional architecture backbone network for feature extraction. The ConvNeXt-tiny pure convolutional architecture backbone network contains four stages of conv, and the sizes of the feature maps obtained in the four different stages are 160×160, 80×80, 40×40, and 20×20 respectively; Construct an intermediate layer feature fusion network to fuse feature information. The intermediate layer fusion network uses add and 1×1 convolution kernels to fuse feature information. In addition to the first stage, the RGB image depth feature map and the HHA image depth feature map obtained by the second stage conv are passed through the intermediate layer feature fusion network and input into layer3 of the CFP centralized pyramid module; The RGB image depth feature map obtained by the third stage conv and the HHA image depth feature map are passed through the intermediate layer feature fusion network and input into layer4 of the CFP centralized pyramid module; The RGB image depth feature map and the HHA image depth feature map obtained in the fourth stage conv are passed through the intermediate layer feature fusion network, and then downsampled by 2 times and input into layer 1 of the CFP centralized pyramid module; The CFP centralized pyramid module fuses the input feature maps of different stages, combines them with the feature data passed in by the previous unit to perform operations, and obtains the final feature representation, thus realizing multimodal data feature fusion.

2. The method for nondestructive detection of litchi quality based on multimodal fusion according to claim 1, wherein: The lightweight network ConvNeXt-tiny feature extraction process is: The lightweight network ConvNeXt-tiny first preprocesses the input image, including a layer normalization LN and a convolution of size 4 and stride 4, with an output channel number of 96; It passes through 4 stages with channel numbers of (96, 192, 384, 768) and corresponding stacking ratios of [3, 3, 9, 3] in sequence. The first stage includes the ConvNeXt Block1 module, the second stage includes the downsampling layer and the ConvNeXt Block2 module, the third stage includes the downsampling layer and the ConvNeXt Block3 module, and the fourth stage includes the downsampling layer and the ConvNeXt Block4 module. The ConvNeXt Block1 module, the ConvNeXt Block2 module, the ConvNeXt Block3 module and the ConvNeXt Block4 module all include a 7×7 depthwise separable convolution layer, two multi-layer perceptrons (MLPs) and two 1×1 convolution layers. The Gaussian error linear unit activation function GELU is used in the two 1×1 convolution layers. The downsampling layer includes an LN layer and a convolution layer with a convolution kernel size of 2 and a stride of 2.

3. The method for nondestructive detection of litchi quality based on multimodal fusion according to claim 1, wherein: Perform image registration on the litchi RGB image and the litchi depth image, specifically including: The pixels of the litchi RGB image are represented as (u R ,v R ,z R ) T , the pixel of the litchi depth image is represented as (u L ,v L ,z L ) T , where u, v, and z represent the horizontal and vertical coordinates of the image and the depth value in the camera coordinate system, respectively, and T represents transposition. The registration formula is as follows: W is a 4×4 matrix expressed as: Among them, RR is the intrinsic parameter matrix of the right camera of the binocular camera calibration, LR is the intrinsic parameter matrix of the left camera of the binocular camera calibration, M is the extrinsic parameter matrix of the optical centers of the two cameras, and r11~r44 are the calculated parameters; Expand the registration formula to register the depth pixel points to the corresponding positions of the RGB pixel points, which can be expressed as: With R ≈z L 。 4. The method for nondestructive detection of litchi quality based on multimodal fusion according to claim 1, wherein: The optimization training process specifically includes: The registered RGB image and HHA map are input into the improved CenterNet network model for synchronous training; Use SGD+Momentum optimizer for target optimization; The DDP mode is adopted and gradient clipping is introduced to prevent gradient explosion.

5. The method for nondestructive detection of litchi quality based on multimodal fusion according to claim 1, wherein: Constructing the Litchi dataset specifically includes: The vehicle uses high-definition cameras and depth cameras to capture multi-angle images of litchi trees in the orchard. Each litchi tree generates multiple sets of RGB and depth images from different angles. By manually screening and labeling litchi RGB images and litchi depth images, litchi without any damage was labeled as healthy litchi, and litchi with obvious damage on the outer surface of the peel was labeled as damaged litchi; Supervised data augmentation is performed on all annotated images. Specific data augmentation methods include random rotation, random flipping, color jittering, color gamut transformation, and noise superposition.

6. The method for nondestructive detection of litchi quality based on multimodal fusion according to claim 1, characterized in that: The encoding of a single-channel litchi depth image into a three-channel HHA image specifically includes: The single-channel Lizhi depth image is encoded into three channels through HHA encoding: horizontal disparity, height above the ground, and norm angle; The three-channel data of horizontal disparity, height above the ground and norm angle are linearly scaled to 0-255 to obtain a three-channel HHA image.

7. A nondestructive detection system for litchi quality based on multimodal fusion, characterized in that: include: Data acquisition module, model building module and detection module; Data acquisition module: used to obtain multiple sets of litchi RGB images and litchi depth images from different angles to construct a litchi dataset; Model building module: used to build an improved CenterNet network model, using a multimodal data fusion strategy; including: CenterNet is used as the basic framework of the litchi quality non-destructive testing model; The lightweight network ConvNeXt-tiny is used as the pure convolutional architecture backbone network of the litchi quality non-destructive testing model; Add a CFP centralized pyramid module at the end of the pure convolutional architecture backbone network; The litchi RGB image and litchi depth image at the same angle are registered with the same pixel width, and the single-channel litchi depth image is encoded into a three-channel HHA image; The registered RGB image and HHA image are used as three-channel inputs, and feature extraction is performed on each of them through a ConvNeXt-tiny pure convolutional architecture backbone network. The two features are fused in the middle layer and input into the CFP centralized pyramid module in a parallel nested manner to achieve multimodal data fusion. Multimodal data fusion specifically includes: The size of the registered RGB image and HHA image is 640×640, and they are respectively input into the ConvNeXt-tiny pure convolutional architecture backbone network for feature extraction. The ConvNeXt-tiny pure convolutional architecture backbone network contains four stages of conv, and the sizes of the feature maps obtained in the four different stages are 160×160, 80×80, 40×40, and 20×20 respectively; Construct an intermediate layer feature fusion network to fuse feature information. The intermediate layer fusion network uses add and 1×1 convolution kernels to fuse feature information. In addition to the first stage, the RGB image depth feature map and the HHA image depth feature map obtained by the second stage conv are passed through the intermediate layer feature fusion network and input into layer3 of the CFP centralized pyramid module; The RGB image depth feature map obtained by the third stage conv and the HHA image depth feature map are passed through the intermediate layer feature fusion network and input into layer4 of the CFP centralized pyramid module; The RGB image depth feature map and the HHA image depth feature map obtained in the fourth stage conv are passed through the intermediate layer feature fusion network, and then downsampled by 2 times and input into layer 1 of the CFP centralized pyramid module; The CFP centralized pyramid module fuses the input feature maps of different stages and performs operations on the feature data passed in by the previous unit to obtain the final feature representation, thus realizing multimodal data feature fusion. Detection module: used to perform distributed training on the improved CenterNet network model based on the Lizhi dataset and optimize the training process; The captured image stream is input into the trained improved CenterNet model to achieve non-destructive testing of litchi quality.

8. A terminal device, characterized in that: include: memory for storing computer programs; as well as A processor, configured to implement the litchi quality nondestructive testing method according to any one of claims 1 to 6 when executing the computer program.

Images