A multi-modal fruit sensory knowledge recognition method, system and storage medium

Abstract

The present application relates to the field of image, especially a multi-modal fruit sensing knowledge recognition method, system and storage medium, multi-scale feature pyramid is obtained through visible light and depth image; the highest layer structure tensor difference matrix is calculated, the mask is generated based on the direction gradient and is fused with the next high visible light feature element by element, and the first modal feature is obtained; Fourier transform is carried out on the first modal feature and the next high depth feature, the latter phase spectrum is reorganized with the former amplitude spectrum to construct cross-modal frequency response matrix, and inverse transform is restored to frequency domain combined feature; the feature space is divided into blocks to construct simplex complex, and the Betti number vector is extracted; the same dimension topological mapping tensor is generated after arrangement, interpolation and mapping, and is multiplied with the original feature point, and the topological perception fusion feature is obtained; the candidate frame is generated, when the information entropy in the frame is greater than the threshold value, the core / edge area is divided and is respectively pooled into response vector, the compensation confidence score is obtained after splicing and mapping, the non-maximum suppression is carried out combined with the confidence score, and fruit recognition and positioning are realized.

Description

Technical Field

[0001] This application belongs to the field of identification, and in particular relates to a multimodal fruit sensing and identification method, system and storage medium. Background Technology

[0002] In real orchard environments, there are interferences from factors such as variable lighting conditions, severe shading by branches and leaves, and dense overlapping of fruits. This paper utilizes a multimodal fruit recognition technology combining visible light and depth RGB-D, where depth images provide spatial location and three-dimensional geometric information of the fruit, while visible light images contain rich texture, color, and surface details. The multimodal recognition method extracts features separately using a two-stream network and then performs cross-modal fusion through channel stitching or bitwise summation.

[0003] However, conventional fusion strategies neglect the fundamental differences between visible light and depth features in spatial gradient and frequency domain, making it difficult to achieve deep modal collaboration. In scenarios involving fruit growth and occlusion, conventional convolutional neural networks rely on local pixel perception, lacking an understanding of the overall spatial topological geometry of the fruit. When the fruit outline is incomplete due to occlusion, it is prone to missed detections due to the loss of topological features. Furthermore, existing methods do not enhance cross-modal features through decoupling and recombining frequency domain amplitude and phase. In the target localization stage, the network typically outputs confidence scores based on global features within candidate bounding boxes. When the information distribution within candidate boxes is chaotic, indiscriminate global pooling can overwhelm information in the core area of ​​the fruit, leading to inaccurate target confidence assessments and, consequently, the false deletion of correctly detected boxes during non-maximum suppression (NMS). Therefore, there is an urgent need for a multimodal recognition method that can deeply fuse cross-modal frequency domain responses, utilize spatial topological invariants, and perform core and edge compensation for high-information-entropy candidate boxes. Summary of the Invention

[0004] To address the issues that existing technologies fail to deeply integrate cross-modal frequency domain responses, fail to utilize spatial topological invariants, and are unable to perform core and edge compensation for high information entropy candidate boxes.

[0005] In the first aspect, the present invention proposes a multimodal fruit perception and recognition method, comprising: The visible light image and depth image of the fruit to be tested are acquired, and multi-scale visible light feature pyramid and multi-scale depth feature pyramid are obtained by network encoding respectively. The structure tensor difference matrix of the highest resolution layer feature in the two pyramids is calculated, the directional gradient magnitude is extracted to generate the highest layer mask, and the first modality feature is obtained by performing a point-by-point multiplication with the second highest resolution visible light feature. A two-dimensional discrete Fourier transform is performed on the first modal feature and the second-highest resolution depth feature. The phase spectrum of the second-highest resolution depth feature and the amplitude spectrum of the first modal feature are combined to construct a cross-mode frequency response matrix. The cross-mode frequency domain merged feature is then obtained by inverse two-dimensional discrete Fourier transform. The cross-mode frequency domain merging features are segmented by a sliding window to construct a simple complex and solve for the corresponding Betti number vector. The Betti number vector is arranged in situ according to the segmentation space and generated into a topology mapping tensor by bilinear interpolation upsampling and pixel-wise convolution channel mapping. The tensor is multiplied by the cross-mode frequency domain merging features using Hadamard multiplication to obtain the topology-aware fusion features. Candidate bounding boxes are generated based on the topology-aware fusion features, and the information distribution entropy of the pixel channels inside each candidate bounding box is calculated. When the information distribution entropy exceeds a preset threshold, it is divided into a core region and an edge region by concentric geometric scaling according to a preset ratio, and global pooling is performed on each region to obtain a core response vector and a boundary response vector. The core response vector and the boundary response vector are concatenated and mapped to output the corresponding target compensation confidence score. Non-maximum suppression is used for screening to complete the identification and localization of the fruit.

[0006] In another aspect, the present invention also proposes a multimodal fruit sensing and recognition system, comprising the following modules: The computation module is used to acquire visible light and depth images of the fruit to be tested, and to obtain multi-scale visible light feature pyramids and multi-scale depth feature pyramids through network encoding, respectively; to calculate the structure tensor difference matrix of the highest resolution layer features in the two pyramids, to extract the directional gradient magnitude to generate the highest layer mask, and to perform a point-by-point multiplication calculation with the second highest resolution visible light features to obtain the first modal features. The construction module is used to perform a two-dimensional discrete Fourier transform on the first modal feature and the second-highest resolution depth feature, combine the phase spectrum of the second-highest resolution depth feature and the amplitude spectrum of the first modal feature to construct a cross-mode frequency response matrix, and obtain the cross-mode frequency domain merged feature through a two-dimensional discrete Fourier inverse transform. The generation module is used to perform sliding window segmentation on the cross-mode frequency domain merging features, construct a simple complex, and solve for the corresponding Betti number vector; the Betti number vector is arranged in situ according to the segmentation space, and a topology mapping tensor is generated by bilinear interpolation upsampling and pixel-wise convolution channel mapping; the tensor is multiplied with the cross-mode frequency domain merging features by Hadamard to obtain the topology-aware fusion features. The filtering module is used to generate candidate bounding boxes based on the topology-aware fusion features, calculate the information distribution entropy of the pixel channels inside each candidate bounding box; when the information distribution entropy exceeds a preset threshold, it is divided into a core region and an edge region by concentric geometric scaling according to a preset ratio, and global pooling is performed on each region to obtain a core response vector and a boundary response vector; the core response vector and the boundary response vector are concatenated and mapped to output the corresponding target compensation confidence score, and non-maximum suppression is used for filtering to complete the identification and localization of the fruit.

[0007] This invention detects subtle abrupt changes in detail between multimodal images by calculating the structural tensor difference matrix of the highest-resolution feature layer and extracting the directional gradient magnitude to generate a mask, guiding the network to focus on key structural regions. It utilizes two-dimensional discrete Fourier transform to reconstruct the phase spectrum of depth features with the amplitude spectrum of visible light features, achieving cross-modal fusion of spatial geometric contours and apparent texture intensity in the frequency domain, thus improving the anti-interference capability of feature extraction. Furthermore, it extracts homogeneous geometric features within spatial segments to construct simple complexes and calculates the Betti number vector, enhancing feature discrimination in overlapping and occlusion scenarios. Finally, it evaluates candidate bounding boxes based on information distribution entropy and compensates confidence scores by extracting response vectors from the core and edge regions through concentric geometric scaling, improving the reliability of multimodal fruit perception, recognition, and localization. Attached Figure Description

[0008] Figure 1 This is a flowchart of the first embodiment. Detailed Implementation

[0009] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0010] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0011] In the first embodiment, the present invention proposes a multimodal fruit perception and recognition method, such as... Figure 1 As shown, it includes: S1. Obtain the visible light image and depth image of the fruit to be tested, and obtain the multi-scale visible light feature pyramid and multi-scale depth feature pyramid through network encoding respectively; calculate the structure tensor difference matrix of the highest resolution layer feature in the two pyramids, extract the directional gradient magnitude to generate the highest layer mask, and perform a point-by-point multiplication calculation with the second highest resolution visible light feature to obtain the first modality feature. A depth vision sensor is used to capture images of the target fruit, obtaining a three-channel visible light color image and a normalized single-channel depth image of the same size and resolution. The visible light image and the depth image are then input into two deep convolutional neural networks. The deep convolutional neural networks adopt a network structure composed of a backbone network and a feature pyramid network. The backbone network performs multi-layer convolution and downsampling pooling to extract preliminary feature maps at different levels. The feature pyramid network enlarges the size of deep features through top-down upsampling operations and adds and fuses them with the corresponding shallow preliminary feature maps pixel by pixel, thereby outputting multi-scale visible light feature pyramids and multi-scale depth feature pyramids with downsampling factors of 4, 8, 16 and 32, respectively. The feature map with a downsampling factor of 4 is extracted as the highest resolution layer feature. The visible light feature map and depth feature map of the highest resolution layer feature are convolved along the horizontal and vertical directions using the Sobel operator to calculate the corresponding first-order partial derivatives. The squared terms and cross-product terms of the first-order partial derivatives of each channel are used to construct a two-dimensional structure tensor matrix reflecting edge and corner features for each pixel. The structure tensor difference matrix is ​​obtained by subtracting the structure tensor matrices of corresponding pixel positions in the multi-scale visible light feature pyramid and the multi-scale depth feature pyramid. The characteristic equation of the structure tensor difference matrix is ​​then solved to obtain... Two non-negative eigenvalues ​​are used. The square root of the sum of the squares of the largest and smallest eigenvalues ​​is used as the directional gradient magnitude value corresponding to each pixel position. The matrix composed of the directional gradient magnitude values ​​of all pixels is mapped to the range of 0 to 1 through max-min normalization to generate the highest layer mask. The highest layer mask is downsampled by 1 / 2 times using the bilinear interpolation algorithm to reduce the spatial dimension by half and align it to the second-highest resolution feature size with a downsampling factor of 8. The highest layer mask after alignment is copied and expanded in the channel dimension and then multiplied by the second-highest resolution visible light feature pixel by pixel to obtain the first modal feature. In an optional embodiment, the step of acquiring the visible light image and depth image of the fruit to be tested, and obtaining the corresponding multi-scale visible light feature pyramid and multi-scale depth feature pyramid through network encoding, includes: The acquired visible light image and depth image are processed to be of uniform size and then input into a dual-branch feature extraction network with residual structure; In the visible light branch and depth branch of the dual-branch feature extraction network, feature maps with different spatial resolutions are extracted layer by layer through multi-stage convolution and downsampling operations. The feature maps output from each stage are arranged in descending order of resolution to construct a multi-scale visible light feature pyramid and a multi-scale depth feature pyramid.

[0012] During size unification processing, the visible light and depth images acquired by the front-end sensor are uniformly scaled to a standard resolution of 640×640 using a bilinear interpolation algorithm. The visible light image is in 3-channel RGB format, and the depth image is in single-channel grayscale or pseudo-color format. The processed images are then synchronously and parallelly input into a two-branch residual feature extraction network with ResNet-50 or an equivalent architecture as its backbone. The visible light and depth branches each have four independent convolutional downsampling stages that do not share weights. Each stage contains multiple combinations of 3×3 residual convolutional blocks with a stride of 1. In the transition between stages, max pooling layers with a stride of 2 or strided convolutions are used for feature downsampling, causing the number of feature map channels to double sequentially from 64, 128, 256, to 512. This results in four layers of feature maps with spatial resolutions of 160×160, 80×80, 40×40, and 20×20 after each stage.

[0013] To avoid idle deep features and fully utilize global semantic information, a top-down Feature Pyramid Network (FPN) structure is employed. Deep high-semantic features with spatial resolutions of 20×20 and 40×40 are upsampled via bilinear interpolation and then fused element-wise with shallow features of 80×80 and 160×160 corresponding channels. The fused features are arranged in descending order of resolution, thus constructing a multi-scale visible light feature pyramid and a multi-scale deep feature pyramid that implicitly incorporate multi-scale global contextual information. Through this mechanism, even explicitly calling only the highest and second-highest resolution layers for computation already includes all semantic parameters of the deep network.

[0014] The dual-branch feature extraction network structure includes a visible light branch and a depth branch with a residual network as the backbone. Each branch contains four convolutional downsampling stages composed of residual convolutional blocks. A feature pyramid network structure for multi-scale semantic information integration is cascaded after feature extraction. The training set consists of paired visible light and depth images containing the fruit to be tested and labeled with real, manually annotated bounding boxes. All image samples are pre-scaled to the same spatial resolution. During the training phase, the connection weight parameters of each convolutional and fusion layer in the entire network are initialized. Image groups from the training set are fed into the forward channel in batches for feature calculation, and the error loss value between the output prediction result and the real, manually annotated bounding boxes is calculated. The backpropagation algorithm is used to solve for the error gradient corresponding to the parameters of each layer of the network. An optimizer is then used to continuously update the overall node weight parameters along the gradient descent direction until the error loss value is reached. The network model training is complete when it converges to a minimal and stable state.

[0015] In an optional embodiment, the step of calculating the structural tensor difference matrix of the highest resolution layer features in the two pyramids, extracting the directional gradient magnitude to generate the highest layer mask, and performing a point-by-point multiplication with the second-highest resolution visible light features to obtain the first modal features includes: Calculate the structure tensor matrices of the visible light features and depth features of the highest resolution layer respectively, and subtract the two to obtain the structure tensor difference matrix; Calculate the gradient magnitude of the structural tensor difference matrix in each direction, assign 1 to pixel positions that are greater than or equal to a preset threshold, and assign 0 to pixel positions that are less than the preset threshold, thereby generating a binarized top-level mask. The spatial size of the highest layer mask is reduced to the same size as the visible light feature of the second highest resolution by a downsampling operation. The downsampled highest layer mask and the visible light feature of the second highest resolution are then multiplied element-wise at their corresponding spatial positions to output the first modal feature.

[0016] For the highest resolution feature layer in the multi-scale feature pyramid, taking a resolution of 160×160 and 64 channels as an example, the feature is reduced to a single channel by taking the mean along the channel dimension, and a Gaussian kernel size of 3×3 and a standard deviation of [missing value]. A smoothing filter is used for preprocessing. The spatial partial derivatives of the feature in the horizontal and vertical directions are calculated using the Sobel operator. and And calculate the inner product components. , and Construct a 2×2 structure tensor matrix for each pixel location; perform the above operation on the two modal features and then perform subtraction to obtain the structure tensor difference matrix that reflects the difference in texture structure.

[0017] Calculate the gradient magnitude of the difference matrix in the orthogonal direction, take the square root of the sum of squares of the differences between the matrix elements, and normalize it to the [0,1] interval. Set a hard threshold, such as 0.65, assigning 1 to pixels with gradient magnitudes greater than or equal to the threshold and 0 to those less than the threshold, generating a 160×160 binarized highest-level mask. To achieve feature size alignment, a 2×2 max-pooling downsampling operation with a stride of 2 is used to halve the mask size to 80×80. Broadcast the downsampled binary mask along the channel dimension, perform element-wise multiplication with the 80×80 sub-high-resolution visible light features, filter out irrelevant background noise, and output the first modality feature tensor.

[0018] S2, Perform a two-dimensional discrete Fourier transform on the first modal feature and the second-highest resolution depth feature, combine the phase spectrum of the second-highest resolution depth feature and the amplitude spectrum of the first modal feature to construct a cross-mode frequency response matrix, and obtain the cross-mode frequency domain merged feature through a two-dimensional discrete Fourier inverse transform. The first modal feature and the sub-high resolution depth feature with a downsampling factor of 8 are sliced ​​and separated according to the channel dimension. For each channel, the Fast Fourier Transform algorithm is used to perform a two-dimensional Discrete Fourier Transform operation to convert the spatial domain feature signal into a frequency domain complex matrix composed of real and imaginary parts. The amplitude spectrum of the complex matrix of each channel of the first modal feature is calculated using the square root of the sum of the squares of the real and imaginary parts of the frequency domain complex matrix. The phase spectrum of the complex matrix of each channel of the sub-high resolution depth feature is calculated using the arctangent function of the ratio of the imaginary to the real part of the frequency domain complex matrix.

[0019] In the frequency domain, the angle information in the phase spectrum of the depth feature is multiplied exponentially with the amplitude spectrum value of the first mode feature using Euler's formula to construct a frequency domain complex matrix that integrates multimodal information as the cross-mode frequency response matrix. The cross-mode frequency response matrix of each channel after recombination is then subjected to a two-dimensional discrete inverse Fourier transform algorithm to convert the feature signal from the frequency domain back to the image space domain. The real part of the inverse transform result is extracted and then spliced ​​and restored according to the original channel order to obtain the cross-mode frequency domain merged feature. In an optional embodiment, the step of performing a two-dimensional discrete Fourier transform on the first modal feature and the second-highest resolution depth feature, constructing a cross-mode frequency response matrix by combining the phase spectrum of the second-highest resolution depth feature and the amplitude spectrum of the first modal feature, and then restoring the cross-mode frequency domain merged feature by a two-dimensional discrete Fourier transform includes: The first mode feature in the spatial domain and the second-highest resolution depth feature are transformed to the frequency domain using a two-dimensional discrete Fourier transform, and the amplitude spectrum of the first mode feature and the phase spectrum of the second-highest resolution depth feature are extracted respectively. The amplitude spectrum of the first modal feature and the phase spectrum of the second-highest resolution depth feature are combined by complex multiplication to generate a cross-mode frequency response matrix with mixed frequency characteristics. Perform a two-dimensional discrete Fourier inverse transform on the cross-mode frequency response matrix to map the matrix from the frequency domain back to the spatial domain, and take the real part as the output cross-mode frequency domain merging feature.

[0020] For the first modal feature tensor and the second-highest resolution depth feature tensor, both with spatial dimensions of 80×80×128, a two-dimensional fast discrete Fourier transform is performed channel-by-channel along the width and height dimensions corresponding to the spatial dimensions, transforming the image features of the two modalities from the spatial domain to the frequency domain. Let the complex form expression of the first modal feature in the frequency domain after the transformation be: According to the formula Extract the first modal feature amplitude spectrum representing the apparent texture of the fruit, where For frequency domain complex numbers The real part, For frequency domain complex numbers The imaginary part; let the frequency domain complex expression of the sub-high resolution depth feature be... According to the formula or The depth feature phase spectrum representing the three-dimensional shape and edge localization of the fruit was extracted. Based on Euler's formula, a complex multiplication combination in polar coordinates was employed. A cross-mode frequency response matrix is ​​synthesized. A two-dimensional discrete Fourier inverse transform is performed on the frequency response matrix channel by channel to decode it back to the spatial domain. The imaginary part caused by the truncation error is discarded, and the real part with the actual response value is retained, resulting in a cross-mode frequency domain merging feature that remains unchanged at a size of 80×80×128.

[0021] S3, the cross-mode frequency domain merging feature is segmented by a sliding window to construct a simple complex and solve for the corresponding Betti number vector; the Betti number vector is arranged in situ according to the segmentation space, and a topology mapping tensor is generated by bilinear interpolation upsampling and pixel-wise convolution channel mapping; the tensor is multiplied by the cross-mode frequency domain merging feature by Hadamard to obtain the topology-aware fusion feature. A sampling window with fixed width and height (both pixels) is set, and the width and height dimensions of the cross-mode frequency domain merged features are successively truncated with a sliding step equal to the window length, resulting in multiple non-overlapping image feature blocks. For each feature matrix obtained from the truncated sliding window, the two-dimensional spatial coordinates of each pixel in the matrix are concatenated with the feature vector at that position to form a three-dimensional spatial point cloud. With each data point as the center, the intersection of spheres is examined to form a Vitris-Lipps simplex complex as the radius parameter gradually increases from zero. The topological invariants of the simplex complex formed within the radius parameter variation range are calculated using the persistent homology algorithm. Only features with a persistent lifetime exceeding a preset threshold are retained. Then, the number of connected components represented by the zero-dimensional homology group and the number of hole loops represented by the one-dimensional homology group are extracted. The second-order and higher Betti numbers, which rarely appear in the two-dimensional image blocks, are ignored to form a one-dimensional Betti number vector.

[0022] All the Betty number vectors calculated within the sliding window are arranged and combined sequentially according to the row and column coordinates of the original spatial segmentation to form a topological feature matrix with reduced spatial resolution and two channels. The topological feature matrix is ​​then enlarged to the same spatial dimensions as the cross-mode frequency domain merging feature using a bilinear interpolation upsampling algorithm. A cross-channel linear combination mapping is then performed using a 1×1 convolution kernel to expand the number of channels in the topological feature matrix to the same number as the cross-mode frequency domain merging feature, thereby generating a topological mapping tensor. The topological mapping tensor is then normalized using a logistic activation function and multiplied element by element with the corresponding position of the cross-mode frequency domain merging feature to extract the topologically aware fusion feature. In an optional embodiment, the step of performing sliding window segmentation on the cross-mode frequency domain merging features, constructing a simplex, and solving for the corresponding Betti number vector includes: A sliding window of fixed size is set, and the cross-mode frequency domain merging features are divided into non-overlapping grids to obtain multiple feature sub-regions; For each feature sub-region, a three-dimensional point composed of the spatial coordinates and feature values ​​of the feature pixels is used as the vertex, and the three-dimensional Euclidean distance between the points is used as the metric to construct a Vitris-Lipps simplex complex. Calculate the topological invariants of the simplex and extract the zeroth-order Betti number, which represents the number of connected components. And the first-order Betty number representing the number of one-dimensional loops. The extracted second-order Betti numbers are concatenated to form a Betti number vector corresponding to the sliding window.

[0023] Using cross-modal frequency domain merged features of dimension 80×80×128 as the input basis, a sliding window with width and height of 8×8 is set in the spatial level. A non-overlapping hard segmentation operation is performed with a step size of 8, dividing the original image into 100 feature sub-regions, each containing 64 128-dimensional feature vectors. For any feature sub-region, the 2D spatial coordinates of each pixel are combined with the L2 norm of the pixel's 128-dimensional feature vector to form a set of 64 3D spatial points. The Euclidean distance between any two vertices in the point set in 3D space is calculated to construct a 64×64 feature distance matrix reflecting semantic topological relationships, thus filling the mathematical metric missing in the original simplex construction. An evolutionary filtering threshold parameter is then used. When the Euclidean distance between two points is less than By connecting points to form edges, and through multi-point interconnection, the high-dimensional simplex is filled, constructing a complete Vitris-Lipps simplex complex. The boundary matrix is ​​calculated using a persistent homology algorithm, and the zeroth-order Betti number with a persistent lifetime exceeding a preset threshold is extracted. First-order Betty number The Betti number feature vector is formed by concatenating one-dimensional arrays. This enables a dimensionality reduction representation of high-dimensional topological structures, and the threshold can be set proportionally based on the maximum value of the feature distance matrix.

[0024] In an optional embodiment, the step of arranging the Betty number vector in situ according to the segmentation space, generating a topology mapping tensor through bilinear interpolation upsampling and pixel-wise convolutional channel mapping, and performing Hadamard dot product on the tensor and the cross-mode frequency domain merged features to obtain topology-aware fusion features includes: The extracted multiple Betty number vectors are rearranged according to their relative positions in the two-dimensional space of the sliding window to which the vectors belong in the original cross-mode frequency domain merged features to construct a low-resolution topological feature map. The low-resolution topological feature map is spatially expanded using a bilinear interpolation algorithm, increasing its size to the same spatial resolution as the cross-mode frequency domain merging feature. The number of channels is then mapped to match the cross-mode frequency domain merging feature through pixel-by-pixel convolution operations, generating a topological mapping tensor. The topology mapping tensor and the cross-mode frequency domain merging feature are subjected to element-wise Hadamard multiplication according to the corresponding channel and spatial position to achieve the fusion of topology features and frequency domain merging features, and the topology-aware fusion feature is output.

[0025] After obtaining 100 Betty number vectors with a dimension of 1×2, a spatial reverse lookup index table is established. This index table is then restored in situ on a two-dimensional layout and stitched together to generate a low-resolution spatial topological feature map of size 10×10 with 2 channels. A bilinear interpolation upsampling algorithm is applied to stretch and enlarge the map by 8 times along its length and width to 80×80 pixels. A pixel-wise convolution with a cascaded kernel of size 1×1 is then applied to increase the channel dimension to 128, and a Sigmoid function is used to restrict the values ​​to the [0,1] interval, generating a topological mapping tensor. Using this tensor as a joint mask, a Hadamard dot product is performed with the backbone base, i.e., the original 80×80×128 cross-mode frequency domain merged features.

[0026] S4. Based on the topology-aware fusion features, candidate bounding boxes are generated, and the information distribution entropy of the pixel channels inside each candidate bounding box is calculated. When the information distribution entropy exceeds a preset threshold, it is divided into a core region and an edge region by concentric geometric scaling according to a preset ratio, and global pooling is performed on each region to obtain a core response vector and a boundary response vector. The core response vector and the boundary response vector are concatenated and mapped to output the corresponding target compensation confidence score. Non-maximum suppression is used for screening to complete the identification and localization of the fruit.

[0027] The topology-aware fusion features are input into the target detection head module, which consists of several layers of convolutional neural networks, to predict the classification probability and regress the bounding box coordinates, thereby generating candidate bounding boxes containing the horizontal and vertical coordinates of the center point and the length and width values ​​of the rectangle.

[0028] The target detection head module typically employs a fully convolutional network structure, consisting of parallel classification and regression sub-networks. Both are based on stacked convolutional layers, such as 3×3 convolutions, with the input being a topology-aware fusion feature map generated upstream. The classification sub-network outputs the class probability of a preset anchor box corresponding to each spatial location through convolution and a sigmoid or softmax activation function. The regression sub-network directly regresses the offset of the anchor box from the true target box, including the center point coordinates and correction values ​​for width and height. During training, paired visible light and depth images are input into the entire model. Using manually labeled bounding boxes and class labels, the weighted sum of classification loss (such as cross-entropy loss or Focal Loss) and regression loss (such as Smooth L1 loss) is calculated as the total error. The weight parameters of the detection head and backbone network are iteratively updated through backpropagation, allowing the predicted boxes to gradually approach the true labels while accurately classifying the fruit target, achieving end-to-end optimization learning.

[0029] For each generated candidate bounding box, the probability distribution density function of the activation values ​​of all pixels covering the feature region within the box in each channel dimension is extracted. The Shannon information entropy is calculated as the information distribution entropy of the pixel channels inside the candidate bounding box. A constant preset threshold is set to measure the lower limit of feature complexity. The calculated information distribution entropy is compared with the preset threshold. When the information distribution entropy of a candidate bounding box is greater than the preset threshold, the absolute position of the horizontal and vertical coordinates of the center point of the current candidate bounding box remains unchanged. The length and width of the bounding box are reduced by a geometric scaling ratio of 50% according to the central symmetry ratio to generate a concentric inner box. The feature map range enclosed by the concentric inner box is defined as the core area, and the feature range of the surrounding ring inside the original candidate bounding box excluding the core area is defined as the edge area. When the information distribution entropy of a candidate bounding box is less than or equal to the preset threshold, the division operation between the core area and the edge area is skipped. The target compensation confidence score is directly set to a constant 1 or the weighting is directly skipped. Only the original classification confidence score generated by the target detection head module is retained to participate in the subsequent non-maximum suppression screening. The spatial global average pooling operation is performed on the feature tensors of the core region and the feature tensors of the edge region along the channel dimension to compress the spatial dimension of each channel, and the one-dimensional core response vector and one-dimensional boundary response vector corresponding to the statistical mean of each channel are calculated and output.

[0030] The core response vector and boundary response vector are concatenated at the end of the feature to form a fused feature vector, which is then input into a multilayer perceptron structure composed of multiple fully connected networks. The output is a scalar value between 0 and 1, which is the corresponding target compensation confidence score. The target compensation confidence score is used as an additional weighting parameter and multiplied by the original classification confidence score predicted by the target detection head module to calculate the target comprehensive score. All candidate bounding boxes are sorted in descending order of target comprehensive score, and the intersection-union ratio (IUGR) overlap area of ​​each box with the highest-scoring box is calculated. Redundant candidate bounding boxes with IUGR overlap areas greater than the suppression threshold are filtered out and deleted by performing non-maximum suppression. The remaining candidate bounding boxes are determined as the fruit targets that have completed the screening, localization, and category recognition.

[0031] The multilayer perceptron (MLP) consists of an input layer, several hidden layers, and an output layer. The input is the concatenation of the core response vector and the boundary response vector. The hidden layers are fully connected and equipped with nonlinear activation functions such as ReLU to extract high-order interaction features. The output layer uses a sigmoid activation function to compress the values ​​to the 0-1 range, generating a target compensation confidence score. During training, the MLP is jointly optimized end-to-end with the entire object detection network. During forward propagation, the compensation score is calculated based on the current features and multiplied by the original classification confidence score output by the detection head to obtain a comprehensive score, which is used for subsequent loss calculation. During backpropagation, the gradient is fed back to each fully connected layer through the comprehensive score, so that its parameters are updated synchronously with the target of the overall detection task, thereby learning the optimal mapping that can adaptively adjust the confidence score according to the response difference between the core region and the edge region.

[0032] To verify the effectiveness of this invention, the hardware environment was configured with a high-performance graphics processor and a large amount of RAM, while the software environment adopted mainstream deep learning frameworks and operating systems. The training dataset consisted of paired visible light images and depth images with manually annotated bounding boxes, and all image samples were pre-scaled to a standard resolution of 640×640 using bilinear interpolation. Mean precision and recall were selected as evaluation metrics. During network model training, the backpropagation algorithm was used to calculate the error gradient, and an optimizer continuously updated the overall node weight parameters along the gradient descent direction until the error loss value was reached. convergence.

[0033] The first set of experiments used only a basic bi-branch feature extraction network and a multi-scale feature pyramid fusion structure, with an average precision of 76.5% and a recall of 73.2%.

[0034] The second group of experiments, building upon the first group, utilized the structure tensor difference mask filtering module, achieving an average precision of 80.1% and a recall of 77.4%.

[0035] The third group of experiments added a two-dimensional discrete Fourier transform cross-mode frequency domain merging module to the second group, with an average precision of 84.3% and a recall of 82.5%.

[0036] The fourth experiment added a topology-aware feature fusion module based on simple complexes to the third experiment, with an average precision of 88.7% and a recall of 87.1%.

[0037] The fifth group of experiments is a complete technical solution that includes all mechanisms. It adds a confidence compensation mechanism based on information distribution entropy on the basis of the fourth group, achieving an average precision of 91.6% and a recall rate of 90.5%.

[0038] The structure tensor difference mask filtering module calculates the structure tensor difference matrix of the highest resolution visible light and depth features in the multi-scale pyramid and extracts the directional gradient magnitude to generate a binarized highest-level mask. After downsampling and alignment, it is calculated by dot product with the second-highest level features. This step filters out irrelevant background noise in the orchard environment, improving the average accuracy by 3.6 percentage points.

[0039] The two-dimensional discrete Fourier transform cross-mode frequency domain merging module transforms features to the frequency domain channel by channel along the channel dimension, and reconstructs the amplitude spectrum representing the fruit's surface texture and the phase spectrum representing the fruit's three-dimensional shape and edge localization by combining them. After the inverse transform is mapped back to the spatial domain, complementary fusion of surface and depth features is achieved, which further improves the average accuracy by 4.2 percentage points.

[0040] The topology-aware feature fusion module based on simplicatures performs spatial sliding window segmentation on the merged features, constructs a Vitris-Lipps simplicature using high-dimensional vectors of internal features, calculates the zeroth and first-order Betti numbers to build a low-resolution topological graph, and then enlarges it to generate a topological mapping tensor. By performing Hadamard dot product operations to suppress discrete noise, this mechanism provides topological continuity guidance in fruit-occluded regions, reduces false negatives, and increases the model recall by 4.6 percentage points.

[0041] The confidence compensation mechanism based on information distribution entropy calculates the information distribution entropy of candidate bounding boxes. When faced with fruit occlusion or interference from branches and leaves causing the entropy value to exceed the limit, it performs concentric geometric scaling to divide the box into core and edge regions and performs global pooling to calculate the target compensation confidence score, which is then combined with the initial score. This strategy corrects local detection biases and, combined with non-maximum suppression, eliminates redundant overlapping boxes, resulting in an overall average accuracy increase of 2.9 percentage points, achieving optimized performance in fruit recognition and localization.

[0042] In a second embodiment, the present invention also proposes a multimodal fruit sensing and recognition system, comprising the following modules: The computation module is used to acquire visible light and depth images of the fruit to be tested, and to obtain multi-scale visible light feature pyramids and multi-scale depth feature pyramids through network encoding, respectively; to calculate the structure tensor difference matrix of the highest resolution layer features in the two pyramids, to extract the directional gradient magnitude to generate the highest layer mask, and to perform a point-by-point multiplication calculation with the second highest resolution visible light features to obtain the first modal features. The construction module is used to perform a two-dimensional discrete Fourier transform on the first modal feature and the second-highest resolution depth feature, combine the phase spectrum of the second-highest resolution depth feature and the amplitude spectrum of the first modal feature to construct a cross-mode frequency response matrix, and obtain the cross-mode frequency domain merged feature through a two-dimensional discrete Fourier inverse transform. The generation module is used to perform sliding window segmentation on the cross-mode frequency domain merging features, construct a simple complex, and solve for the corresponding Betti number vector; the Betti number vector is arranged in situ according to the segmentation space, and a topology mapping tensor is generated by bilinear interpolation upsampling and pixel-wise convolution channel mapping; the tensor is multiplied with the cross-mode frequency domain merging features by Hadamard to obtain the topology-aware fusion features. The filtering module is used to generate candidate bounding boxes based on the topology-aware fusion features, calculate the information distribution entropy of the pixel channels inside each candidate bounding box; when the information distribution entropy exceeds a preset threshold, it is divided into a core region and an edge region by concentric geometric scaling according to a preset ratio, and global pooling is performed on each region to obtain a core response vector and a boundary response vector; the core response vector and the boundary response vector are concatenated and mapped to output the corresponding target compensation confidence score, and non-maximum suppression is used for filtering to complete the identification and localization of the fruit.

[0043] In an optional embodiment, the step of acquiring the visible light image and depth image of the fruit to be tested, and obtaining the corresponding multi-scale visible light feature pyramid and multi-scale depth feature pyramid through network encoding, includes: The acquired visible light image and depth image are processed to be of uniform size and then input into a dual-branch feature extraction network with residual structure; In the visible light branch and depth branch of the dual-branch feature extraction network, feature maps with different spatial resolutions are extracted layer by layer through multi-stage convolution and downsampling operations. The feature maps output from each stage are arranged in descending order of resolution to construct a multi-scale visible light feature pyramid and a multi-scale depth feature pyramid.

[0044] In an optional embodiment, the step of calculating the structural tensor difference matrix of the highest resolution layer features in the two pyramids, extracting the directional gradient magnitude to generate the highest layer mask, and performing a point-by-point multiplication with the second-highest resolution visible light features to obtain the first modal features includes: Calculate the structure tensor matrices of the visible light features and depth features of the highest resolution layer respectively, and subtract the two to obtain the structure tensor difference matrix; Calculate the gradient magnitude of the structural tensor difference matrix in each direction, assign 1 to pixel positions that are greater than or equal to a preset threshold, and assign 0 to pixel positions that are less than the preset threshold, thereby generating a binarized top-level mask. The spatial size of the highest layer mask is reduced to the same size as the visible light feature of the second highest resolution by a downsampling operation. The downsampled highest layer mask and the visible light feature of the second highest resolution are then multiplied element-wise at their corresponding spatial positions to output the first modal feature.

[0045] In an optional embodiment, the step of performing a two-dimensional discrete Fourier transform on the first modal feature and the second-highest resolution depth feature, constructing a cross-mode frequency response matrix by combining the phase spectrum of the second-highest resolution depth feature and the amplitude spectrum of the first modal feature, and then restoring the cross-mode frequency domain merged feature by a two-dimensional discrete Fourier transform includes: The first mode feature in the spatial domain and the second-highest resolution depth feature are transformed to the frequency domain using a two-dimensional discrete Fourier transform, and the amplitude spectrum of the first mode feature and the phase spectrum of the second-highest resolution depth feature are extracted respectively. The amplitude spectrum of the first modal feature and the phase spectrum of the second-highest resolution depth feature are combined by complex multiplication to generate a cross-mode frequency response matrix with mixed frequency characteristics. Perform a two-dimensional discrete Fourier inverse transform on the cross-mode frequency response matrix to map the matrix from the frequency domain back to the spatial domain, and take the real part as the output cross-mode frequency domain merging feature.

[0046] In an optional embodiment, the step of performing sliding window segmentation on the cross-mode frequency domain merging features, constructing a simplex, and solving for the corresponding Betti number vector includes: A sliding window of fixed size is set, and the cross-mode frequency domain merging features are divided into non-overlapping grids to obtain multiple feature sub-regions; For each feature sub-region, a three-dimensional point composed of the spatial coordinates and feature values ​​of the feature pixels is used as the vertex, and the three-dimensional Euclidean distance between the points is used as the metric to construct a Vitris-Lipps simplex complex. Calculate the topological invariants of the simplex and extract the zeroth-order Betti number, which represents the number of connected components. And the first-order Betty number representing the number of one-dimensional loops. The extracted second-order Betti numbers are concatenated to form a Betti number vector corresponding to the sliding window.

[0047] In an optional embodiment, the step of arranging the Betty number vector in situ according to the segmentation space, generating a topology mapping tensor through bilinear interpolation upsampling and pixel-wise convolutional channel mapping, and performing Hadamard dot product on the tensor and the cross-mode frequency domain merged features to obtain topology-aware fusion features includes: The extracted multiple Betty number vectors are rearranged according to their relative positions in the two-dimensional space of the sliding window to which the vectors belong in the original cross-mode frequency domain merged features to construct a low-resolution topological feature map. The low-resolution topological feature map is spatially expanded using a bilinear interpolation algorithm, increasing its size to the same spatial resolution as the cross-mode frequency domain merging feature. The number of channels is then mapped to match the cross-mode frequency domain merging feature through pixel-by-pixel convolution operations, generating a topological mapping tensor. The topology mapping tensor and the cross-mode frequency domain merging feature are subjected to element-wise Hadamard multiplication according to the corresponding channel and spatial position to achieve the fusion of topology features and frequency domain merging features, and the topology-aware fusion feature is output.

[0048] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0049] The functional modules shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0050] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0051] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0052] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A multimodal fruit sensing and recognition method, characterized in that, include: The visible light image and depth image of the fruit to be tested are acquired, and multi-scale visible light feature pyramid and multi-scale depth feature pyramid are obtained by network encoding, respectively; Calculate the structural tensor difference matrix of the highest resolution layer features in the two pyramids, extract the directional gradient magnitude to generate the highest layer mask, and perform a point-by-point multiplication with the second highest resolution visible light features to obtain the first modal features; A two-dimensional discrete Fourier transform is performed on the first modal feature and the second-highest resolution depth feature. The phase spectrum of the second-highest resolution depth feature and the amplitude spectrum of the first modal feature are combined to construct a cross-mode frequency response matrix. The cross-mode frequency domain merged feature is then obtained by inverse two-dimensional discrete Fourier transform. The cross-mode frequency domain merging features are segmented by a sliding window to construct a simple complex and solve for the corresponding Betti number vector. The Betti number vector is arranged in situ according to the segmentation space and generated into a topology mapping tensor by bilinear interpolation upsampling and pixel-wise convolution channel mapping. The tensor is multiplied by the cross-mode frequency domain merging features using Hadamard multiplication to obtain the topology-aware fusion features. Candidate bounding boxes are generated based on the topology-aware fusion features, and the information distribution entropy of the pixel channels inside each candidate bounding box is calculated. When the information distribution entropy exceeds a preset threshold, it is divided into a core region and an edge region by concentric geometric scaling according to a preset ratio, and global pooling is performed on each region to obtain a core response vector and a boundary response vector. The core response vector and the boundary response vector are concatenated and mapped to output the corresponding target compensation confidence score. Non-maximum suppression is used for screening to complete the identification and localization of the fruit.

2. The method according to claim 1, characterized in that, The process of acquiring visible light and depth images of the fruit to be tested, and then encoding them via a network to obtain corresponding multi-scale visible light feature pyramids and multi-scale depth feature pyramids, includes: The acquired visible light image and depth image are processed to be of uniform size and then input into a dual-branch feature extraction network with residual structure; In the visible light branch and depth branch of the dual-branch feature extraction network, feature maps with different spatial resolutions are extracted layer by layer through multi-stage convolution and downsampling operations. The feature maps output from each stage are arranged in descending order of resolution to construct a multi-scale visible light feature pyramid and a multi-scale depth feature pyramid.

3. The method according to claim 1, characterized in that, The structure tensor difference matrix for calculating the highest resolution layer features in the two pyramids is used to extract the directional gradient magnitude to generate the highest layer mask. This mask is then multiplied by the second-highest resolution visible light features to obtain the first modality features, including: Calculate the structure tensor matrices of the visible light features and depth features of the highest resolution layer respectively, and subtract the two to obtain the structure tensor difference matrix; Calculate the gradient magnitude of the structural tensor difference matrix in each direction, assign 1 to pixel positions that are greater than or equal to a preset threshold, and assign 0 to pixel positions that are less than the preset threshold, thereby generating a binarized top-level mask. The spatial size of the highest layer mask is reduced to the same size as the visible light feature of the second highest resolution by a downsampling operation. The downsampled highest layer mask and the visible light feature of the second highest resolution are then multiplied element-wise at their corresponding spatial positions to output the first modal feature.

4. The method according to claim 3, characterized in that, The step of performing a two-dimensional discrete Fourier transform on the first modal feature and the second-highest resolution depth feature, combining the phase spectrum of the second-highest resolution depth feature and the amplitude spectrum of the first modal feature to construct a cross-mode frequency response matrix, and then restoring it through a two-dimensional discrete Fourier inverse transform to obtain the cross-mode frequency domain merged feature, includes: The first mode feature in the spatial domain and the second-highest resolution depth feature are transformed to the frequency domain using a two-dimensional discrete Fourier transform, and the amplitude spectrum of the first mode feature and the phase spectrum of the second-highest resolution depth feature are extracted respectively. The amplitude spectrum of the first modal feature and the phase spectrum of the second-highest resolution depth feature are combined by complex multiplication to generate a cross-mode frequency response matrix with mixed frequency characteristics. Perform a two-dimensional discrete Fourier inverse transform on the cross-mode frequency response matrix to map the matrix from the frequency domain back to the spatial domain, and take the real part as the output cross-mode frequency domain merging feature.

5. The method according to claim 1, characterized in that, The process of performing sliding window segmentation on the cross-mode frequency domain merging features, constructing a simplex, and solving for the corresponding Betti number vector includes: A sliding window of fixed size is set, and the cross-mode frequency domain merging features are divided into non-overlapping grids to obtain multiple feature sub-regions; For each feature sub-region, a Vitris-Lipps simplex is constructed using feature pixels as vertices and feature distances between adjacent pixels as connecting edges. Calculate the topological invariants of the simplex and extract the zeroth-order Betti number, which represents the number of connected components. And the first-order Betty number representing the number of one-dimensional loops. The extracted second-order Betti numbers are concatenated to form a Betti number vector corresponding to the sliding window.

6. The method according to claim 5, characterized in that, The process involves arranging the Betty number vectors in situ according to the segmentation space, upsampling via bilinear interpolation, and generating a topological mapping tensor through pixel-wise convolutional channel mapping. Then, the tensor is multiplied with the cross-modal frequency domain merged features using Hadamard multiplication to obtain topologically aware fusion features, including: The extracted multiple Betty number vectors are rearranged according to their relative positions in the two-dimensional space of the sliding window to which the vectors belong in the original cross-mode frequency domain merged features to construct a low-resolution topological feature map. The low-resolution topological feature map is spatially expanded using a bilinear interpolation algorithm, increasing its size to the same spatial resolution as the cross-mode frequency domain merging feature. The number of channels is then mapped to match the cross-mode frequency domain merging feature through pixel-by-pixel convolution operations, generating a topological mapping tensor. The topology mapping tensor and the cross-mode frequency domain merging feature are subjected to element-wise Hadamard multiplication according to the corresponding channel and spatial position to achieve the fusion of topology features and frequency domain merging features, and the topology-aware fusion feature is output.

7. A multimodal fruit sensing and recognition system, characterized in that, Includes the following modules: The computation module is used to acquire visible light and depth images of the fruit to be tested, and to obtain multi-scale visible light feature pyramids and multi-scale depth feature pyramids through network encoding, respectively; to calculate the structure tensor difference matrix of the highest resolution layer features in the two pyramids, to extract the directional gradient magnitude to generate the highest layer mask, and to perform a point-by-point multiplication calculation with the second highest resolution visible light features to obtain the first modal features. The construction module is used to perform a two-dimensional discrete Fourier transform on the first modal feature and the second-highest resolution depth feature, combine the phase spectrum of the second-highest resolution depth feature and the amplitude spectrum of the first modal feature to construct a cross-mode frequency response matrix, and obtain the cross-mode frequency domain merged feature through a two-dimensional discrete Fourier inverse transform. The generation module is used to perform sliding window segmentation on the cross-mode frequency domain merging features, construct a simple complex, and solve for the corresponding Betti number vector; the Betti number vector is arranged in situ according to the segmentation space, and a topology mapping tensor is generated by bilinear interpolation upsampling and pixel-wise convolution channel mapping; the tensor is multiplied with the cross-mode frequency domain merging features by Hadamard to obtain the topology-aware fusion features. The filtering module is used to generate candidate bounding boxes based on the topology-aware fusion features, calculate the information distribution entropy of the pixel channels inside each candidate bounding box; when the information distribution entropy exceeds a preset threshold, it is divided into a core region and an edge region by concentric geometric scaling according to a preset ratio, and global pooling is performed on each region to obtain a core response vector and a boundary response vector; the core response vector and the boundary response vector are concatenated and mapped to output the corresponding target compensation confidence score, and non-maximum suppression is used for filtering to complete the identification and localization of the fruit.

8. The system according to claim 7, characterized in that, The process of acquiring visible light and depth images of the fruit to be tested, and then encoding them via a network to obtain corresponding multi-scale visible light feature pyramids and multi-scale depth feature pyramids, includes: The acquired visible light image and depth image are processed to be of uniform size and then input into a dual-branch feature extraction network with residual structure; In the visible light branch and depth branch of the dual-branch feature extraction network, feature maps with different spatial resolutions are extracted layer by layer through multi-stage convolution and downsampling operations. The feature maps output from each stage are arranged in descending order of resolution to construct a multi-scale visible light feature pyramid and a multi-scale depth feature pyramid.

9. The system according to claim 7, characterized in that, The structure tensor difference matrix for calculating the highest resolution layer features in the two pyramids is used to extract the directional gradient magnitude to generate the highest layer mask. This mask is then multiplied by the second-highest resolution visible light features to obtain the first modality features, including: Calculate the structure tensor matrices of the visible light features and depth features of the highest resolution layer respectively, and subtract the two to obtain the structure tensor difference matrix; Calculate the gradient magnitude of the structural tensor difference matrix in each direction, assign 1 to pixel positions that are greater than or equal to a preset threshold, and assign 0 to pixel positions that are less than the preset threshold, thereby generating a binarized top-level mask. The spatial size of the highest layer mask is reduced to the same size as the visible light feature of the second highest resolution by a downsampling operation. The downsampled highest layer mask and the visible light feature of the second highest resolution are then multiplied element-wise at their corresponding spatial positions to output the first modal feature.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when executed by a processor, implements the method as described in any one of claims 1-6.

Images