A fruit zero-shot spoilage detection method based on multi-modal structure inversion

Abstract

The present application provides a fruit zero-sample decay detection method based on multi-modal structure inversion. By extracting surface geometric parameters and local surface roughness parameters, a curvature compensation factor and a roughness attenuation term are constructed to correct the standard Fresnel reflection coefficient. The signal attenuation caused by external geometry and skin roughness is pre-removed, and the physical dielectric characteristics are extracted from the non-planar biological tissue. Then, cross-modal adaptive fusion is realized through visual anchoring cross-attention and dynamic gating weighting, and the diagnostic weights of physical and visual modalities are dynamically adjusted according to the decay stage, covering both internal and surface priority decay modes. Through classification confidence discrimination, accurate grading of known varieties and automatic diversion of unknown varieties are realized. For unknown varieties, the use of synthetic exclusive health threshold combined with asymmetric fusion decision significantly improves the accuracy and reliability of non-contact fruit quality detection in real unstructured agricultural scenarios.

Description

Technical Field

[0001] This invention relates to the field of non-destructive testing technology, and more specifically, to a zero-sample fruit spoilage detection method based on multimodal structure inversion. Background Technology

[0002] With the rapid evolution of intelligent agricultural supply chains and modern cold chain warehousing and logistics technologies, high-throughput, fully automated, and non-destructive internal quality assessment and freshness grading of high-value agricultural products have become core aspects of ensuring food safety and improving refined quality control. However, traditional computer vision-based automated inspection methods mainly rely on industrial cameras to capture external images of fruits, which are essentially "surface layer" perceptions. Many high-value fruits (such as apples, peaches, and oranges) often exhibit a significant "internal-priority" spoilage pattern during post-harvest storage—that is, the core internal tissues have undergone cell membrane collapse, tissue browning, and abnormal accumulation of free water, while the outer peel still maintains its intact color. In such scenarios, pure visual inspection systems, due to the physical barrier of the epidermis, cannot effectively capture subsurface and deep lesions, resulting in a very high false negative rate, making it difficult to meet the needs of modern supply chains for precise interception of hidden spoilage.

[0003] To address the limitations of purely visual methods, academia and industry have attempted to introduce chemometric techniques such as near-infrared spectroscopy and hyperspectral imaging, utilizing the energy resonance between photons in specific wavelengths and internal water molecules or sugars to estimate internal quality. However, these optical sensing methods still face numerous challenges in practical deployment: Firstly, hyperspectral data is massive and hardware costs are high; it is highly sensitive to fluctuations in ambient light and easily interfered with by non-pathological noise such as uneven waxy layers on the fruit peel, making large-scale deployment across environments and batches difficult. Secondly, while radio frequency and millimeter-wave sensing technologies offer advantages in high penetration and low cost, enabling non-contact sensing of changes in microscopic dielectric properties (such as water migration), existing moisture inversion architectures are generally based on the simplified assumption of electromagnetic scattering on a "flat surface." In reality, various fruits and vegetables possess complex non-planar macroscopic curvatures; when millimeter waves strike curved surfaces, they induce severe nonlinear scattering and significant geometric phase errors. This intense electromagnetic distortion deeply intertwines the external geometry of fruits and vegetables with their internal microscopic dielectric properties, making it extremely difficult for ordinary inversion algorithms to extract the weak dielectric constant drift that is heavily masked by geometric noise and indicates early, hidden decay.

[0004] To address the aforementioned multimodal collaboration and cross-target detection challenges, existing research often attempts to train radio frequency (RF) rating systems within a highly constrained closed-set supervised learning paradigm. However, these methods typically only achieve improvements within a single species or specific dimension, failing to simultaneously balance detection accuracy and cross-species generalization. Obtaining high-quality supervised data often requires extensive manual feature labeling using destructive instruments such as penetrating probes, leading to models being highly susceptible to overfitting to specific fruit category RF features. When the system encounters entirely new fruit species outside the training distribution, detection accuracy often suffers a catastrophic drop. Although pre-trained visual language models have made significant progress in zero-shot inference in recent years, their semantic knowledge bases lack conceptual understanding of "deep physical anomalies beneath a flawless surface" due to deeply ingrained visual morphological biases. Forcibly aligning invisible RF physical features with the text space of a large model with rigid symmetry can easily induce severe semantic mismatches and feature collapse. Currently, the industry lacks a universal physical sensing architecture that can effectively compensate for physical distortions caused by macroscopic geometry and break through the limitations of single-modal sensing, so as to meet the need for zero-sample universal spoilage classification of heterogeneous and unknown fruits in unstructured agricultural scenarios. Summary of the Invention

[0005] The technical problem to be solved by this invention is how to solve the problems of missed detection of hidden fruit decay and the difficulty of generalizing detection models to unknown species.

[0006] This invention provides a method for detecting zero-sample spoilage of fruit based on multimodal structure inversion, comprising the following steps: S1, synchronously acquire visual images and radio frequency reflection signals of the target fruit; obtain the measured amplitude based on the radio frequency reflection signals; S2, perform semantic segmentation on the visual image, extract the two-dimensional boundary of the target fruit region, project it onto the radar coordinate system and construct a three-dimensional ellipsoid model to extract surface geometric parameters, and calculate local surface roughness parameters within the target fruit region; S3, based on surface geometric parameters and local surface roughness parameters, construct curvature compensation factor and roughness attenuation term to correct the pre-acquired standard Fresnel reflection coefficient, and obtain the generalized reflection coefficient; S4. A forward propagation model is constructed based on the generalized reflection coefficient. Nonlinear optimization is performed with the measured amplitude as the target, and the physical dielectric characteristics are obtained by solving the solution. S5 performs cross-modal fusion of physical dielectric features and high-dimensional features of visual images, and generates fused features and intermediate gating states through visually anchored cross-attention and dynamic gating weighting. S6. Input the fused features into the classification head to obtain the classification confidence. If the classification confidence is not lower than the preset threshold, determine that the target fruit is a known variety and output the spoilage level; otherwise, the target fruit is an unknown variety and extract the deep physical anomaly descriptor from the intermediate gating state. S7, based on deep physical anomaly descriptors and visual images, uses a pre-trained visual language model to obtain visual health confidence and synthesizes a unique health threshold for unknown varieties, calculates physical anomaly scores, and performs asymmetric fusion decision-making by combining visual health confidence to output a corruption level prediction.

[0007] Compared with existing technologies, this application has the following advantages: By performing semantic segmentation and three-dimensional ellipsoidal model fitting on visual images, the geometric morphology information in the visual images is converted into surface geometric parameters and local surface roughness parameters. Then, based on the surface geometric parameters and local surface roughness parameters, a curvature compensation factor and a roughness attenuation term are constructed to correct the standard Fresnel reflection coefficient. This pre-extracts the signal attenuation caused by external geometry and epidermal roughness, extracting morphology-independent physical dielectric features from non-planar biological tissues. Subsequently, cross-modal adaptive fusion is achieved through visually anchored cross-attention and dynamic gating weighting, and the physical modality is dynamically adjusted according to the putrefaction stage. The diagnostic weights of the visual modality cover both internal-priority and surface-priority spoilage modes. Furthermore, classification confidence scores enable precise grading of known varieties and automatic triage of unknown varieties. For unknown varieties, a synthesized, proprietary health threshold, combined with asymmetric fusion decision-making, significantly improves the accuracy and reliability of non-contact fruit quality detection in real-world, unstructured agricultural scenarios. Moreover, this application represents a paradigm shift from closed-set supervised classification to open-set zero-sample physical sensing, significantly enhancing the accuracy, robustness, and generalization ability of non-contact fruit spoilage detection in real-world, unstructured agricultural scenarios without requiring the re-collection of radio frequency training data for unknown varieties.

[0008] In one possible implementation, S1 specifically includes: S11 employs a co-located RGB camera and radar on the same bracket, maintaining relative pose and pre-calibrating the camera intrinsic parameter matrix and camera-radar extrinsic parameter transformation matrix; S12, the target fruit is placed on a standardized testing platform. At a fixed working distance, the radar continuously transmits millimeter waves at a preset frame rate and receives radio frequency reflected signals. The high signal-to-noise ratio range-amplitude profile is generated by time-domain averaging and aggregation as the measured amplitude at each range gate. Simultaneously, a static high-resolution image is synchronously acquired by the RGB camera as a visual image, forming a synchronous data pair containing a visual image and a radar range-amplitude profile for each group.

[0009] Compared with existing technologies, by co-locating RGB cameras and millimeter-wave radars and pre-calibrating intrinsic and extrinsic parameter matrices, strict consistency between visual images and radio frequency reflection signals in spatial reference is ensured, eliminating cross-modal spatial mismatch caused by sensor pose deviation; through synchronous trigger acquisition mechanism, strict alignment between visual images and radio frequency signals in the time dimension is ensured, avoiding feature misalignment caused by time asynchrony.

[0010] In one possible implementation, S2 specifically includes: S21, Input the visual image into the lightweight semantic segmentation network, extract the target fruit region on the visual image, and obtain the two-dimensional boundary of the target fruit region on the visual image. The two-dimensional boundary is composed of several boundary pixels. S22 uses the camera intrinsic parameter matrix and the camera-radar extrinsic parameter transformation matrix to transform each boundary pixel on the two-dimensional boundary to the three-dimensional coordinates under the radar coordinate system, generating a three-dimensional point set to characterize the spatial distribution of the fruit surface. S23. The three-dimensional point set is fitted with a standard triaxial ellipsoid centered at the origin of the radar coordinate system using least squares fitting. The optimization objective is to minimize the sum of squared geometric distances from each three-dimensional coordinate to the surface of the triaxial ellipsoid. The optimal semi-axis lengths of the triaxial ellipsoid in the X, Y, and Z axes are obtained, and thus the three-dimensional ellipsoid model is obtained. S24. Let the central axis of the radar beam be a spatial ray extending from the radar transmission center along the beam pointing direction. The direction of this spatial ray is the radar incident direction. Substitute the parametric equation of the spatial ray into the three-dimensional ellipsoid model, solve for all intersection points between the spatial ray and the three-dimensional ellipsoid model, and select the effective intersection point located in front of the radar illumination and closest to it as the actual illumination point of the radar beam on the surface of the target fruit. Solve for the spatial gradient of the three-dimensional ellipsoid model at the actual illumination point and normalize it to obtain the surface normal vector. The angle between the surface normal vector and the radar incident direction is calculated as the radar incident angle. Differential geometry calculations are used to calculate the two local principal curvature radii at the actual irradiation point based on the optimal semi-axis lengths in the X, Y, and Z axes and the actual irradiation point. and ; The surface normal vector, radar incident angle, and two local principal curvature radii. and Surface geometric parameters; S25, calculate the variance of the image texture gradient within the target fruit region as a local surface roughness parameter to quantify the effect of the micro-texture of the target fruit peel on the diffuse scattering of electromagnetic wave reflection.

[0011] Compared with existing technologies, this method uses least squares fitting to transform the discrete three-dimensional point set parameters into a three-dimensional ellipsoid model. By using an algorithm to find the intersection of a spatial ray and the ellipsoid, the actual illumination point of the radar beam on the curved surface is accurately calculated. Then, differential geometry is used to accurately solve for the surface normal vector, incident angle, local principal curvature radius, and local surface roughness parameters used to quantify the influence of the micro-texture of the target fruit peel on the electromagnetic wave reflection and diffuse scattering.

[0012] In one possible implementation, S3 specifically includes: S31, based on the operating wavelength of the radar beam Distance from the actual illumination point to the radar Local principal curvature radius and Construct curvature compensation factor ; According to the radar incident angle Local surface roughness parameters and operating wavelength Construct roughness attenuation term ; S32, Pre-acquiring standard Fresnel reflection coefficient The standard Fresnel reflection coefficient is corrected using a curvature compensation factor and a roughness attenuation term. The generalized reflection coefficient is obtained as follows: ; In the formula, Represents the complex permittivity; Compared with existing technologies, this method achieves normalized compensation for the effective scattering cross section of curved surfaces by constructing a curvature compensation factor. It accurately models and mathematically compensates for the echo energy divergence caused by the curvature change of the fruit surface, so that the generalized reflection coefficient no longer includes the influence component of external geometric shape. By constructing a Rayleigh roughness attenuation term, the diffuse scattering weakening effect of surface micro-roughness on millimeter wave coherent reflection is quantified. The final generalized reflection coefficient is a pure physical quantity that removes the dual interference of external geometric shape and surface roughness, solving the deep-seated problem of geometric noise masking weak dielectric drift in existing technologies.

[0013] In one possible implementation, S4 specifically includes: S41, Constructing a forward propagation model based on the generalized reflection coefficient: ; In the formula, This indicates the one-way propagation distance of the radar to the actual illumination point. This is the reference value for the transmitted amplitude calibrated at the factory of the radar. This represents the geometrical diffusion attenuation of spherical waves. This represents the absorption and attenuation factor of electromagnetic waves propagating in lossy media inside fruits. This indicates taking the imaginary part of a complex number; S42, using the measured amplitude as the target value and the complex permittivity as the variable to be optimized, establishes a nonlinear optimization objective function that minimizes the sum of squared residuals between the measured amplitude and the forward propagation model, and solves for the physical dielectric characteristics: ; In the formula, Indicates physical dielectric characteristics, Indicates the first Measured amplitude.

[0014] Compared with existing technologies, this method establishes a complete physical mapping relationship from the complex permittivity to the radar received amplitude by constructing a forward propagation model; it achieves an accurate inversion solution for the complex permittivity by performing nonlinear optimization iteration with the measured amplitude as the target; and it obtains physical dielectric characteristics. The real part reflects the dielectric constant of the fruit tissue (related to water content), while the imaginary part reflects the dielectric loss (related to ion concentration and free water content). Together, they constitute a complete physical fingerprint characterizing the internal physiological state.

[0015] In one possible implementation, S5 specifically includes: S51, input the visual image into the pre-trained ResNet backbone network to extract high-dimensional features; S52 maps high-dimensional features and physical dielectric features to a unified embedding dimension to obtain an aligned visual sequence. and physical sequence , Represents the real number field. Represents a real matrix with L rows and D columns; S53, with visual sequence Through learnable weight matrix Generate query vectors With physical sequence Through learnable weight matrix and Generate key vectors respectively Sum value vector ; query vector Key vector Sum value vector Substituting into the scaling dot product attention formula: ; Obtain a cross-modal symptom map that retrieves relevant information from a physical sequence using visual anchoring; where, Indicates the dimension of the query vector key vector; S54, visual sequence and cross-modal symptom diagram After concatenation along the channel dimension, the input is given to a gated neural network and then activated by a Sigmoid activation function to obtain a dynamic gate vector. : ; In the formula, This represents the bias term of the gated neural network; S55, based on dynamic gating vector Cross-modal symptom mapping through cross-attention and visual sequences The fused features are obtained by adaptive weighted summation. : ; At the same time, the dynamic gating vector With cross-modal symptom map The element-wise product state serves as an intermediate gating state.

[0016] Compared with existing technologies, the high-dimensional features extracted by the ResNet backbone network and the physical dielectric features obtained by radar inversion are mapped to a unified embedding space through a projection head, which solves the problem of incommensurability of heterogeneous modal features in mathematical space. Through the visual anchoring cross-attention mechanism, information related to the visual sequence in the physical sequence is selectively retrieved and weighted and aggregated, forming a cross-modal spatial correspondence between the epidermal region and the internal dielectric state. Through the adaptive weighted summation of the dynamic gating vector g, the fused features can be dynamically adjusted according to different stages of decay (early stage (no visual abnormalities) biased towards the physical modality, late stage (obvious visual symptoms) biased towards the visual modality), realizing the adaptive allocation of diagnostic weights and significantly improving the adaptability of the fused features to different stages of decay and different varieties.

[0017] In one possible implementation, S6 specifically includes: S61 will fuse features The input is aggregated by global average pooling and then fed into a supervised classification head consisting of a fully connected layer and Softmax. The output is the probability distribution of the target fruit belonging to each known level of spoilage, and the maximum probability value is taken as the classification confidence. S62, if the classification confidence is greater than or equal to a preset threshold, the current target fruit is determined to be a known variety, and the spoilage level corresponding to the highest probability is output as the spoilage level of the target fruit; otherwise, if the classification confidence is less than the preset threshold, the current target fruit is determined to be an unknown variety, and a deep physical anomaly descriptor is extracted based on the intermediate gating state. : ; In the formula, This represents the L2 norm.

[0018] Compared with existing technologies, by inputting fused features into the supervised classification head and obtaining classification confidence, the system achieves automatic sorting between known and unknown varieties using a preset threshold. This enables the system to adaptively switch processing paths in an open world—directly outputting accurate spoilage levels for known varieties and triggering zero-shot inference branches for unknown varieties. This avoids the catastrophic misjudgments caused by traditional classifiers forcibly classifying unseen varieties. At the same time, it extracts deep physical anomaly descriptors from the intermediate gating states of unknown varieties.

[0019] In one possible implementation, S7 uses a pre-trained visual language model to obtain visual health confidence based on deep physical anomaly descriptors and visual images, and synthesizes a specific health threshold for unknown varieties, specifically including: First, the visual image is fed into the frozen CLIP image encoder to obtain image embedding vectors. Simultaneously, K structured cue words for known varieties are input into the CLIP text encoder to obtain text embedding vectors. The cosine similarity between the image embedding vector and each text embedding vector is then calculated. : ; In the formula, Represents the image embedding vector. Indicates the first Text embedding vectors of known varieties; After Softmax normalization, the semantic similarity vector is obtained. : ; Next, retrieve the pre-stored health benchmarks for each known variety. ; Finally, based on semantic similarity vectors Weighted interpolation is performed on the health benchmarks of known varieties to synthesize the specific health thresholds for unknown varieties. : ; Compared with existing technologies, this method extracts the semantic similarity between images of unknown fruits and textual descriptions of known varieties by using a frozen CLIP pre-trained visual language model. It utilizes the rich visual-semantic association knowledge contained in the large-scale pre-trained model to establish species proximity relationships between unknown varieties and known varieties. By using semantic similarity vectors to perform weighted interpolation on the health benchmarks of known varieties to synthesize exclusive thresholds, the physical rationale is that the more similar the visual appearance of varieties, the closer their intrinsic basic physiological parameters (such as benchmark water content) are. Therefore, mixing the health benchmarks of known varieties based on similarity is a reasonable estimate of the health status of unknown varieties.

[0020] In one possible implementation, S7 calculates a physical anomaly score, combines it with visual health confidence for asymmetric fusion decision-making, and outputs a corruption level prediction, specifically including: First, based on deep physical anomaly descriptors and exclusive health thresholds Calculate the physical anomaly score : ; In the formula, Indicates the preset scale parameters; Secondly, the visual images were matched with positive and negative cue words using CLIP image-text matching, and the visual health confidence scores were obtained after Softmax normalization. ; Next, the physical anomaly score will be calculated. and visual appearance health confidence level Perform asymmetric fusion decision-making to calculate the probability of corruption. : ; In the formula, For physical weighting coefficients, For competition-triggered gain coefficient, To modify the activation function of the linear unit; Finally, based on the corruption probability within each preset corruption level range, the corruption level of the unknown species is determined and output.

[0021] Compared with existing technologies, this method maps the magnitude of anomalies to a probability space in the (0,1) interval by using physical anomaly scores, thus achieving a probabilistic measurement of the intensity of physical anomalies. It obtains visual health confidence through CLIP image-text matching, thus achieving a quantitative assessment of flawless appearance. It achieves competitive fusion of physical anomaly scores and visual confidence through an asymmetric fusion formula, generating positive activation only when physical anomalies significantly exceed visual confidence. When the fruit appears intact on the outside but is severely rotten inside, the physical judgment has the highest priority, forcibly overturning optimistic predictions based on visual appearance, and effectively intercepting visual omissions caused by internally prioritized rot. Attached Figure Description

[0022] Figure 1 This is a flowchart of this application. Detailed Implementation

[0023] First, those skilled in the art should understand that these embodiments are merely used to explain the technical principles of the embodiments of this application and are not intended to limit the scope of protection of the embodiments of this application. Those skilled in the art can make adjustments as needed to adapt to specific application scenarios.

[0024] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of this application based on the specific circumstances.

[0025] In the embodiments of this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0026] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0027] See Figure 1 As shown, this application discloses a method for detecting zero-sample spoilage of fruit based on multimodal structure inversion. Before executing the method of this application, the emission amplitude reference value is calibrated: a standard metal reflector (with a known radar cross-section) is placed at the standard working distance on the test platform, the radar emits millimeter waves and receives the radio frequency reflected signal, the echo amplitude is measured and substituted into the radar equation to deduce the emission amplitude reference value. ; The method of this application includes the following steps: S1, synchronously acquire visual images and radio frequency reflection signals of the target fruit; obtain the measured amplitude based on the radio frequency reflection signal; specifically including: S11 employs a co-located RGB camera and radar on the same bracket, maintaining relative pose and pre-calibrating the camera intrinsic parameter matrix and camera-radar extrinsic parameter transformation matrix; The camera intrinsic parameter matrix A 3×3 upper triangular matrix is ​​defined as: ; In the formula, , These are the equivalent pixel focal lengths in the X and Y directions, respectively. , These are the pixel coordinates of the visual image; for the intrinsic parameter matrix... Inverting the matrix yields the intrinsic inverse matrix, which is used to back-project the pixel homogeneous coordinates onto the camera normalization plane. ; Camera-radar extrinsic transformation matrix A rigid transformation from the camera coordinate system to the radar coordinate system is achieved by concatenating a 3×3 rotation matrix R and a 3×1 translation vector t. .

[0028] S12, the target fruit is placed on a standardized testing platform. At a fixed working distance, the radar continuously transmits millimeter waves at a preset frame rate and receives radio frequency reflected signals. The high signal-to-noise ratio range-amplitude profile is generated by time-domain averaging and aggregation as the measured amplitude at each range gate. Simultaneously, a static high-resolution image is synchronously acquired by the RGB camera as a visual image, forming a synchronous data pair containing a visual image and a radar range-amplitude profile for each group. In this embodiment, during data acquisition, the vertical working distance between the sensor plane and the fruit support surface is fixed at 30 cm; the radar continuously transmits frequency-modulated continuous wave signals at a frame rate of 20 Hz and receives radio frequency reflected signals. After acquiring approximately 30 frames of data, they are time-domain averaged and aggregated into a single frame of high signal-to-noise ratio range-amplitude profile, which serves as the range gate. Measured amplitude at the location , This represents the number of doors at a distance.

[0029] S2 performs semantic segmentation on the visual image, extracts the two-dimensional boundary of the target fruit region, projects it onto the radar coordinate system, and constructs a three-dimensional ellipsoidal model to extract surface geometric parameters and calculate local surface roughness parameters within the target fruit region; specifically including: S21, input the visual image into a lightweight semantic segmentation network to extract the target fruit region from the visual image and obtain the two-dimensional boundary of the target fruit region on the visual image. The two-dimensional boundary is composed of several boundary pixels, denoted as . , This represents the total number of boundary pixels.

[0030] S22 uses the camera intrinsic parameter matrix and the camera-radar extrinsic parameter transformation matrix to transform each boundary pixel on the two-dimensional boundary to three-dimensional coordinates in the radar coordinate system. The expression is as follows: ; Generate a three-dimensional point set to characterize the spatial distribution of fruit surface. .

[0031] S23. The three-dimensional point set is fitted with a standard triaxial ellipsoid centered at the origin of the radar coordinate system using least squares fitting. The optimization objective is to minimize the sum of squared geometric distances from each three-dimensional coordinate to the surface of the triaxial ellipsoid. The optimal semi-axis lengths of the triaxial ellipsoid in the X, Y, and Z axes are obtained, and thus the three-dimensional ellipsoid model is obtained. The equation of the standard triaxial ellipsoid is: ; In the formula, These are the semi-axis lengths in the X, Y, and Z directions, respectively; The optimization objective is to minimize the sum of squared geometric distances from each 3D coordinate to the surface of the triaxial ellipsoid. ; This application employs the Levenberg-Marquardt algorithm to iteratively solve the aforementioned nonlinear least squares problem, obtaining the optimal semi-axis length. This leads to the construction of a three-dimensional ellipsoid model.

[0032] S24, Let the central axis of the radar beam be a spatial ray extending from the radar transmission center along the beam pointing direction, and the direction of this spatial ray be the radar incident direction. The parametric equation of the space ray Substituting into the three-dimensional ellipsoid model, the expression is: ; In the formula, This indicates a globally uniform scaling scalar. Represent the three-dimensional increment vector; solve for all intersections between the spatial ray and the three-dimensional ellipsoid model. The effective intersection point located in front of and closest to the radar beam is selected as the actual illumination point of the radar beam on the surface of the target fruit. The spatial gradient of the three-dimensional ellipsoidal model at the actual illumination point is solved and normalized to obtain the surface normal vector: ; in, ; Calculate the surface normal vector relative to radar incident direction The angle between them is used as the radar incident angle. : ; Differential geometry calculations are used to calculate the two local principal curvature radii at the actual irradiation point based on the optimal semi-axis lengths in the X, Y, and Z axes and the actual irradiation point. and ; The surface normal vector, radar incident angle, and two local principal curvature radii. and Surface geometric parameters; S25, calculate the variance of the image texture gradient within the target fruit region as a local surface roughness parameter: ; In the formula, and These are the gradients in the horizontal and vertical directions, respectively; the variance of the Sobel gradient magnitudes of all pixels within this region is calculated and used as a local surface roughness parameter. : ; Local surface roughness parameters Used to quantify the effect of epidermal microtexture on diffuse scattering of electromagnetic wave reflection—the coarser the texture, the more diffuse the scattering. The larger the value, the more significant the diffuse scattering loss.

[0033] S3, based on surface geometric parameters and local surface roughness parameters, constructs a curvature compensation factor and a roughness attenuation term to correct the pre-acquired standard Fresnel reflection coefficient, thus obtaining the generalized reflection coefficient; specifically including: S31, based on the operating wavelength of the radar beam Distance from the actual illumination point to the radar Local principal curvature radius and Construct curvature compensation factor The physical meaning of the curvature compensation factor is to normalize the effective scattering cross section reduced due to the curvature of the target fruit surface: when the target fruit surface is flat, The curvature compensation factor takes the limit value. 1; When the surface of the target fruit is severely curved, the curvature compensation factor is less than 1, which is used to accurately compensate for the weakening of the echo caused by the curvature of the shape. According to the radar incident angle Local surface roughness parameters and operating wavelength Construct roughness attenuation term The roughness attenuation term is used to compensate for the diffuse scattering loss caused by the roughness of the skin. When the target fruit skin is smooth ( When the curvature compensation factor approaches 1, there is no attenuation; when the target fruit skin is rough, the curvature compensation factor is less than 1, compensating for the coherent reflection loss caused by diffuse scattering.

[0034] S32, Pre-acquiring standard Fresnel reflection coefficient : ; The standard Fresnel reflection coefficient describes the variation of the amplitude and phase of the reflection coefficient of vertically polarized electromagnetic waves with the incident angle and the dielectric constant of the medium under oblique incidence. It is a classic formula in radar scattering and electromagnetic propagation modeling. The standard Fresnel reflection coefficient is corrected using a curvature compensation factor and a roughness attenuation term. The generalized reflection coefficient is obtained as follows: ; In the formula, Represents the complex permittivity; After this correction, the signal attenuation caused by the target's external geometry (macroscopic surface curvature) and surface micro-roughness in the generalized reflection coefficient has been pre-exposed, retaining only the reflection information related to the internal dielectric properties.

[0035] S4. A forward propagation model is constructed based on the generalized reflection coefficient. Nonlinear optimization is performed with the measured amplitude as the target, and the physical dielectric characteristics are obtained by solving the solution. Based on the theory of electromagnetic wave propagation in lossy media, the medium wavenumber is defined. Its imaginary part The absorption attenuation coefficient of the medium. The imaginary part of the complex number is used to describe the energy loss caused by moisture and ion conduction when electromagnetic waves propagate inside the target. According to the theory of spherical wave radiation, the amplitude of electromagnetic waves attenuates inversely with the square of the distance during propagation in free space; that is, the geometrical diffusion factor is... ,in This is the one-way propagation distance of the radar to the actual irradiation point.

[0036] S41, a forward propagation model is constructed based on the generalized reflection coefficient, specifically: The millimeter waves emitted by the radar travel over distance After spherical geometric diffusion at the point of origin, the energy reaches the target surface, and a portion of the energy is reflected by the generalized reflection coefficient. The surface reflection described is directly returned to the radar, forming the surface reflection component; another part of the energy is transmitted into the interior of the fruit, where it is absorbed and attenuated before being reflected by the internal tissue and transmitted back to the surface to the radar, forming the internal reflection component; the two components are coherently superimposed at the radar receiver.

[0037] For corruption detection scenarios dominated by internal reflection, the theoretical receiver amplitude model can be expressed as: ; In the formula, This indicates the one-way propagation distance of the radar to the actual illumination point. This is the reference value for the transmitted amplitude calibrated at the factory of the radar. This represents the geometrical diffusion attenuation of spherical waves. This represents the absorption and attenuation factor of electromagnetic waves propagating in lossy media inside fruits. This indicates taking the imaginary part of a complex number; S42, using the measured amplitude as the target value and the complex permittivity as the variable to be optimized, establishes a nonlinear optimization objective function that minimizes the sum of squared residuals between the measured amplitude and the forward propagation model, and solves for the physical dielectric characteristics: ; In the formula, Indicates physical dielectric characteristics, Indicates the first Measured amplitude.

[0038] This application employs a gradient descent optimization algorithm (such as BFGS or L-BFGS) for iterative solution to obtain the physical dielectric characteristics. , , For the real part, The real part reflects changes in moisture content, while the imaginary part reflects the accumulation of free water and changes in ion concentration. Together, they indicate the degree of putrefaction.

[0039] S5 performs cross-modal fusion of physical dielectric features and high-dimensional features of visual images, generating fused features and intermediate gating states through visually anchored cross-attention and dynamic gating weighting; specifically including: S51, input the visual image into the pre-trained ResNet backbone network to extract high-dimensional features; in this embodiment of the application, the pre-trained ResNet-50 backbone network takes the feature map output by its last convolutional layer and transforms it into high-dimensional features through global average pooling. The high-dimensional features encode external visual attributes such as fruit peel color distribution, surface texture, and fungal patches.

[0040] S52 maps high-dimensional features and physical dielectric features to a unified embedding dimension to obtain an aligned visual sequence. and physical sequence , Represents the real number field. Represents a real matrix with L rows and D columns; In this embodiment, high-dimensional features and physical dielectric features are fed into their respective projection heads. Each projection head consists of two learnable linear layers (including layer normalization and ReLU activation), mapping both to a unified embedding dimension. The aligned visual sequence is obtained. and physical sequence Since this embodiment does not involve spatial location unfolding, the sequence length... .

[0041] S53, with visual sequence Through learnable weight matrix Generate query vectors With physical sequence Through learnable weight matrix and Generate key vectors respectively Sum value vector ; query vector Key vector Sum value vector Substituting into the scaling dot product attention formula: ; A cross-modal symptom map was obtained by retrieving relevant information from the physical sequence using vision as the anchor condition. Using visual features as the anchor condition, the internal dielectric information most relevant to the current visual observation area was sparsely retrieved from the physical sequence, and an explicit correspondence between the epidermal region and the dielectric state below was established.

[0042] S54, visual sequence and cross-modal symptom diagram After concatenation along the channel dimension, the input is given to a gated neural network and then activated by a Sigmoid activation function to obtain a dynamic gate vector. : ; In the formula, This represents the bias term of the gated neural network; The Sigmoid function compresses the output to the (0,1) interval; a dynamically gated vector. The j-th dimension being close to 1 indicates that the physical modality should dominate in this feature dimension, while being close to 0 indicates that the visual modality should dominate.

[0043] S55, based on dynamic gating vector Cross-modal symptom mapping through cross-attention and visual sequences The fused features are obtained by adaptive weighted summation. : ; At the same time, the dynamic gating vector With cross-modal symptom map The element-wise product state serves as an intermediate gating state.

[0044] S6, input the fused features into the classification head to obtain the classification confidence score. If the classification confidence score is not lower than a preset threshold, the target fruit is determined to be a known variety and the spoilage level is output; otherwise, the target fruit is an unknown variety, and a deep physical anomaly descriptor is extracted from the intermediate gating state; specifically including: S61 will fuse features The input is aggregated by global average pooling and then fed into a supervised classification head consisting of a fully connected layer and a Softmax layer. The output is the probability distribution of the target fruit belonging to each known level of spoilage. , This represents the total number of corruption levels, with the highest probability value among them. As the classification confidence level, the corruption level corresponding to the classification confidence level is output; in the embodiments of this application, the corruption levels include Level 1 corruption: intact, Level 2 corruption: slight corruption, Level 3 corruption: moderate corruption, and Level 4 corruption: severe corruption.

[0045] S62, if the classification confidence is greater than or equal to a preset threshold, the current target fruit is determined to be a known variety, and the spoilage level corresponding to the highest probability is output as the spoilage level of the target fruit; otherwise, if the classification confidence is less than the preset threshold, the current target fruit is determined to be an unknown variety, and a deep physical anomaly descriptor is extracted based on the intermediate gating state. : ; In the formula, This represents the L2 norm.

[0046] In this embodiment of the application, the preset threshold is 0.6; if The target fruit currently belongs to a known variety in the training set; If the target fruit is an unknown variety outside the training distribution, then the current target fruit belongs to that variety.

[0047] S7, based on deep physical anomaly descriptors and visual images, uses a pre-trained visual language model to obtain visual health confidence and synthesizes a unique health threshold for unknown varieties, calculates physical anomaly scores, and performs asymmetric fusion decision-making by combining visual health confidence to output a corruption level prediction.

[0048] Specifically, based on deep physical anomaly descriptors and visual images, a pre-trained visual language model is used to obtain visual appearance health confidence and synthesize a unique health threshold for unknown varieties, including: First, the visual image is fed into the frozen CLIP image encoder to obtain image embedding vectors. Simultaneously, K structured cue words for known varieties are input into the CLIP text encoder to obtain text embedding vectors. The cosine similarity between the image embedding vector and each text embedding vector is then calculated. : ; In the formula, Represents the image embedding vector. Indicates the first Text embedding vectors of known varieties; After Softmax normalization, the semantic similarity vector is obtained. : ; Next, retrieve the pre-stored health benchmarks for each known variety. ; Finally, based on semantic similarity vectors Weighted interpolation is performed on the health benchmarks of known varieties to synthesize the specific health thresholds for unknown varieties. : .

[0049] In S7, physical anomaly scores are calculated, and asymmetric fusion decision is performed by combining visual appearance health confidence scores to output a corruption level prediction, specifically including: First, based on deep physical anomaly descriptors and exclusive health thresholds Calculate the physical anomaly score : ; In the formula, Indicates the preset scale parameters; Secondly, the visual images were matched with positive and negative cue words using CLIP image-text matching, and the visual health confidence scores were obtained after Softmax normalization. ; Next, the physical anomaly score will be calculated. and visual appearance health confidence level Perform asymmetric fusion decision-making to calculate the probability of corruption. : ; In the formula, For physical weighting coefficients, For competition-triggered gain coefficient, To modify the activation function of the linear unit; It is a purely visually driven probability of decay, suitable for scenarios where there are already obvious signs of decay on the surface; The base probability is purely driven by physics (α is 0.70 in this embodiment), and the basic influence of physical anomalies on the final judgment is modulated by the physical weight coefficient. As a competition trigger (β is 0.40 in this embodiment), through The function performs directional activation on the difference between the physical anomaly score and the visual confidence score, generating positive activation only when the physical anomaly score is strictly greater than the visual confidence score, ensuring that the veto is only triggered in conflict scenarios.

[0050] The asymmetric fusion decision-making in this application can be summarized into three typical scenarios: when all health is consistent. ≈0.95 ≈0.05, (1− )≈0.05 and =0, P≈0.05 is considered intact; uniform decay ≈0.05 ≈0.95, (1− )≈0.95 and Positive activation, P≈1 indicates severe corruption; internal priority corruption occurs when... ≈0.92 (outermost condition) ≈0.90 (internal severe anomaly), (1− )≈0.08, ( )≈0.37, because - ≈−0.02<0, therefore =0, P=max(0.08,0.37)=0.37 is judged as slight to moderate putrefaction, successfully intercepting visual misses; when Increase further until exceeding At that time, the third sub-item is activated to provide an additional safety margin and enhance interception capabilities.

[0051] Finally, the corruption level of the unknown variety is determined and output based on the corruption probability falling within each preset corruption level range; in this embodiment, the preset corruption level ranges are: Level 1 corruption: ; Second-degree corruption: ; Third-level corruption: ; Level 4 corruption: .

[0052] In the description of the embodiments of this application, it should be noted that the terms "inner" and "outer" and other terms indicating direction or positional relationship are based on the direction or positional relationship shown in the drawings. This is only for the convenience of description and does not indicate or imply that the device or component must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this application.

[0053] In the description of this application, the references to terms such as "an embodiment," "some embodiments," "in this embodiment," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0054] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting zero-sample spoilage of fruit based on multimodal structure inversion, characterized in that, Includes the following steps: S1, synchronously acquire visual images and radio frequency reflection signals of the target fruit; obtain the measured amplitude based on the radio frequency reflection signals; S2, perform semantic segmentation on the visual image, extract the two-dimensional boundary of the target fruit region, project it onto the radar coordinate system and construct a three-dimensional ellipsoid model to extract surface geometric parameters, and calculate local surface roughness parameters within the target fruit region; S3, based on surface geometric parameters and local surface roughness parameters, construct curvature compensation factor and roughness attenuation term to correct the pre-acquired standard Fresnel reflection coefficient, and obtain the generalized reflection coefficient; S4. A forward propagation model is constructed based on the generalized reflection coefficient. Nonlinear optimization is performed with the measured amplitude as the target, and the physical dielectric characteristics are obtained by solving the solution. S5 performs cross-modal fusion of physical dielectric features and high-dimensional features of visual images, and generates fused features and intermediate gating states through visually anchored cross-attention and dynamic gating weighting. S6. Input the fused features into the classification head to obtain the classification confidence. If the classification confidence is not lower than the preset threshold, determine that the target fruit is a known variety and output the spoilage level. Otherwise, if the target fruit is of an unknown variety, extract the deep physical anomaly descriptor from the intermediate gating state; S7, based on deep physical anomaly descriptors and visual images, uses a pre-trained visual language model to obtain visual health confidence and synthesizes a unique health threshold for unknown varieties, calculates physical anomaly scores, and performs asymmetric fusion decision-making by combining visual health confidence to output a corruption level prediction.

2. The method for detecting zero-sample spoilage of fruit based on multimodal structure inversion according to claim 1, characterized in that, S1 specifically includes: S11 employs a co-located RGB camera and radar on the same bracket, maintaining relative pose and pre-calibrating the camera intrinsic parameter matrix and camera-radar extrinsic parameter transformation matrix; S12, the target fruit is placed on a standardized testing platform. At a fixed working distance, the radar continuously transmits millimeter waves at a preset frame rate and receives radio frequency reflected signals. The high signal-to-noise ratio range-amplitude profile is generated by time-domain averaging and aggregation as the measured amplitude at each range gate. Simultaneously, a static high-resolution image is synchronously acquired by the RGB camera as a visual image, forming a synchronous data pair containing a visual image and a radar range-amplitude profile for each group.

3. The method for detecting zero-sample spoilage of fruit based on multimodal structure inversion according to claim 2, characterized in that, S2 specifically includes: S21, Input the visual image into the lightweight semantic segmentation network, extract the target fruit region on the visual image, and obtain the two-dimensional boundary of the target fruit region on the visual image. The two-dimensional boundary is composed of several boundary pixels. S22 uses the camera intrinsic parameter matrix and the camera-radar extrinsic parameter transformation matrix to transform each boundary pixel on the two-dimensional boundary to the three-dimensional coordinates under the radar coordinate system, generating a three-dimensional point set to characterize the spatial distribution of the fruit surface. S23. The three-dimensional point set is fitted with a standard triaxial ellipsoid centered at the origin of the radar coordinate system using least squares fitting. The optimization objective is to minimize the sum of squared geometric distances from each three-dimensional coordinate to the surface of the triaxial ellipsoid. The optimal semi-axis lengths of the triaxial ellipsoid in the X, Y, and Z axes are obtained, and thus the three-dimensional ellipsoid model is obtained. S24. Let the central axis of the radar beam be a spatial ray extending from the radar transmission center along the beam pointing direction. The direction of this spatial ray is the radar incident direction. Substitute the parametric equation of the spatial ray into the three-dimensional ellipsoid model, solve for all intersection points between the spatial ray and the three-dimensional ellipsoid model, and select the effective intersection point located in front of the radar illumination and closest to it as the actual illumination point of the radar beam on the surface of the target fruit. Solve for the spatial gradient of the three-dimensional ellipsoid model at the actual illumination point and normalize it to obtain the surface normal vector. The angle between the surface normal vector and the radar incident direction is calculated as the radar incident angle. Differential geometry calculations are used to calculate the two local principal curvature radii at the actual irradiation point based on the optimal semi-axis lengths in the X, Y, and Z axes and the actual irradiation point. and ; The surface normal vector, radar incident angle, and two local principal curvature radii. and Surface geometric parameters; S25, calculate the variance of the image texture gradient within the target fruit region as a local surface roughness parameter to quantify the effect of the micro-texture of the target fruit peel on the diffuse scattering of electromagnetic wave reflection.

4. The method for detecting zero-sample spoilage of fruit based on multimodal structure inversion according to claim 3, characterized in that, S3 specifically includes: S31, based on the operating wavelength of the radar beam Distance from the actual illumination point to the radar Local principal curvature radius and Construct curvature compensation factor ; According to the radar incident angle Local surface roughness parameters and operating wavelength Construct roughness attenuation term ; S32, Pre-acquiring standard Fresnel reflection coefficient The standard Fresnel reflection coefficient is corrected using a curvature compensation factor and a roughness attenuation term. The generalized reflection coefficient is obtained as follows: ； In the formula, This represents the complex permittivity.

5. The method for detecting zero-sample spoilage of fruit based on multimodal structure inversion according to claim 4, characterized in that, S4 specifically includes: S41, Constructing a forward propagation model based on the generalized reflection coefficient: ； In the formula, This indicates the one-way propagation distance of the radar to the actual illumination point. This is the reference value for the transmitted amplitude calibrated at the factory of the radar. This represents the geometrical diffusion attenuation of spherical waves. This represents the absorption and attenuation factor of electromagnetic waves propagating in lossy media inside fruits. This indicates taking the imaginary part of a complex number; S42, using the measured amplitude as the target value and the complex permittivity as the variable to be optimized, establishes a nonlinear optimization objective function that minimizes the sum of squared residuals between the measured amplitude and the forward propagation model, and solves for the physical dielectric characteristics: ； In the formula, Indicates physical dielectric characteristics, Indicates the first Measured amplitude.

6. The method for detecting zero-sample spoilage of fruit based on multimodal structure inversion according to claim 5, characterized in that, S5 specifically includes: S51, input the visual image into the pre-trained ResNet backbone network to extract high-dimensional features; S52 maps high-dimensional features and physical dielectric features to a unified embedding dimension to obtain an aligned visual sequence. and physical sequence , Represents the real number field. Represents a real matrix with L rows and D columns; S53, with visual sequence Through learnable weight matrix Generate query vectors With physical sequence Through learnable weight matrix and Generate key vectors respectively Sum value vector ; query vector Key vector Sum value vector Substituting into the scaling dot product attention formula: ； Obtain a cross-modal symptom map that retrieves relevant information from a physical sequence using visual anchoring; where, Indicates the dimension of the query vector key vector; S54, visual sequence and cross-modal symptom diagram After concatenation along the channel dimension, the input is given to a gated neural network and then activated by a Sigmoid activation function to obtain a dynamic gate vector. : ； In the formula, This represents the bias term of the gated neural network; S55, based on dynamic gating vector Cross-modal symptom mapping through cross-attention and visual sequences The fused features are obtained by adaptive weighted summation. : ； At the same time, the dynamic gating vector With cross-modal symptom map The element-wise product state serves as an intermediate gating state.

7. The method for detecting zero-sample spoilage of fruit based on multimodal structure inversion according to claim 6, characterized in that, S6 specifically includes: S61 will fuse features The input is aggregated by global average pooling and then fed into a supervised classification head consisting of a fully connected layer and Softmax. The output is the probability distribution of the target fruit belonging to each known level of spoilage, and the maximum probability value is taken as the classification confidence. S62, if the classification confidence is greater than or equal to a preset threshold, the current target fruit is determined to be a known variety, and the spoilage level corresponding to the highest probability is output as the spoilage level of the target fruit; otherwise, if the classification confidence is less than the preset threshold, the current target fruit is determined to be an unknown variety, and a deep physical anomaly descriptor is extracted based on the intermediate gating state. : ； In the formula, This represents the L2 norm.

8. The method for detecting zero-sample spoilage of fruit based on multimodal structure inversion according to claim 7, characterized in that, S7 utilizes deep physical anomaly descriptors and visual images, employs a pre-trained visual language model to obtain visual appearance health confidence, and synthesizes a unique health threshold for unknown varieties. Specifically, this includes: First, the visual image is fed into the frozen CLIP image encoder to obtain image embedding vectors. Simultaneously, K structured cue words for known varieties are input into the CLIP text encoder to obtain text embedding vectors. The cosine similarity between the image embedding vector and each text embedding vector is then calculated. : ； In the formula, Represents the image embedding vector. Indicates the first Text embedding vectors of known varieties; After Softmax normalization, the semantic similarity vector is obtained. : ； Next, retrieve the pre-stored health benchmarks for each known variety. ; Finally, based on semantic similarity vectors Weighted interpolation is performed on the health benchmarks of known varieties to synthesize the specific health thresholds for unknown varieties. : 。 9. The method for detecting zero-sample spoilage of fruit based on multimodal structure inversion according to claim 8, characterized in that, In S7, physical anomaly scores are calculated, and asymmetric fusion decision is performed by combining visual appearance health confidence scores to output a corruption level prediction, specifically including: First, based on deep physical anomaly descriptors and exclusive health thresholds Calculate the physical anomaly score : ； In the formula, Indicates the preset scale parameters; Secondly, the visual images were matched with positive and negative cue words using CLIP image-text matching, and the visual health confidence scores were obtained after Softmax normalization. ; Next, the physical anomaly score will be calculated. and visual appearance health confidence level Perform asymmetric fusion decision-making to calculate the probability of corruption. : ； In the formula, For physical weighting coefficients, For competition-triggered gain coefficient, To modify the activation function of the linear unit; Finally, based on the corruption probability within each preset corruption level range, the corruption level of the unknown species is determined and output.

Images