Multimodal sensing and intelligent decision-making method for detecting embedded stones in potatoes

Abstract

This invention discloses a method and system for detecting embedded stones in potatoes using multimodal sensing and intelligent decision-making, specifically relating to the field of non-destructive testing and sorting of agricultural products. The invention involves triggering two sensors to acquire X-ray penetration images and ultrasonic echo signals when potatoes enter the detection area in a single column. The raw data is then preprocessed to generate standardized feature maps. Next, the preprocessed X-ray images and ultrasonic feature maps are spatiotemporally registered and input into a trained multi-channel deep learning fusion decision network. This network extracts deep features from both modalities in parallel, performs adaptive weighted fusion at the feature layer, outputs the classification probability of whether each potato contains embedded stones, and performs pixel-level localization of the stone locations. Finally, the intelligent decision-making network's judgment result and localization information are sent to the sorting execution mechanism in real time, removing potatoes containing embedded stones from the production line.

Description

Technical Field

[0001] This invention relates to the field of non-destructive testing and sorting technology for agricultural products, and more specifically, to a method for detecting embedded stones in potatoes using multimodal sensing and intelligent decision-making. Background Technology

[0002] During mechanized harvesting, potatoes easily become embedded with small stones collected from the field (hereinafter referred to as "embedded stones"). These defects are highly concealed and cannot be detected by traditional optical sorting equipment, posing a long-standing quality and safety hazard in the potato processing industry (such as potato chip and French fry production). Existing solutions have the following main shortcomings: X-ray single-mode detection: Although it can penetrate the interior of objects, it has limited ability to distinguish between the internal moisture and density changes of potatoes and stones (mainly composed of silicates). Especially when the density of stones and potatoes is similar, the signal-to-noise ratio is low, which can easily lead to missed or false detections.

[0003] Visible light / near-infrared detection: can only detect surface defects and is completely ineffective for embedded stones.

[0004] Manual sampling inspection: inefficient, labor-intensive, and unreliable.

[0005] Therefore, there is an urgent need for a detection technology and equipment that has high accuracy, strong adaptability, and can meet the needs of high-speed industrial sorting. Summary of the Invention

[0006] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a method for detecting embedded stones in potatoes using multimodal sensing and intelligent decision-making, in order to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: S1: The potato to be tested passes through the detection area in a single row, and the X-ray imaging unit and the high-frequency ultrasonic scanning unit are triggered simultaneously to acquire its X-ray penetration image and ultrasonic echo signal, respectively. S2: Denoise and contrast enhancement of X-ray images; Time-frequency analysis of ultrasonic echo signals to extract time-domain and frequency-domain feature maps reflecting internal structural discontinuities; S3: The preprocessed X-ray image and ultrasonic feature map are spatiotemporally registered and input into a trained multi-channel deep learning fusion decision network. This network extracts deep features of the two modalities in parallel, performs adaptive weighted fusion at the feature layer, outputs the classification probability of whether each potato contains embedded stones, and performs pixel-level localization of the stone positions; S4: The judgment results and positioning information output by the intelligent decision network are sent to the sorting execution mechanism in real time to remove potatoes containing embedded stones from the production line.

[0008] Preferably, in step S1, the variable frequency speed control conveyor belt is started, and the potatoes to be tested are arranged in a single row and uniformly placed into the detection area through the vibration sorting device and the spacing limit plate at the entrance. In the detection area, X-ray imaging unit and high-frequency ultrasonic scanning unit are deployed in parallel. When the potato arrives at the starting point of the detection area, multiple photoelectric encoders detect the leading edge position in real time and immediately generate a synchronous trigger pulse signal. After the pulse signal is received by the central synchronous controller, it will drive the two sensing units simultaneously.

[0009] Preferably, in step S2, after acquiring the synchronously collected raw multimodal data, the system performs a preprocessing process in parallel to transform the raw multimodal data into standardized feature representations. In X-ray image preprocessing, the raw images received by the X-ray detector typically contain system noise, scattering noise, and background fluctuations; the following steps are used for processing: Background correction and noise reduction: First, perform dark field correction: from the original image Subtracting the dark field image without X-rays To eliminate detector dark current noise; Gain normalization is then performed using a standard reference image. Perform division operations to correct the non-uniformity of the detector pixel response; Corrected image The specific calculation method is as follows: ; Finally, an adaptive nonlocal mean filtering denoising algorithm is applied to effectively filter out random noise while preserving the edge details of the stones and tissues. In contrast and detail enhancement, to highlight the contrast between the stones (usually manifested as local bright areas) and the surrounding tissue, a contrast-limited adaptive histogram equalization algorithm is adopted. This algorithm divides the image into several sub-regions, performs histogram equalization on each, and prevents excessive amplification of noise by limiting contrast, effectively enhancing subtle density difference information.

[0010] Finally, the preprocessed X-ray feature image is obtained. Its pixel value directly reflects the relative attenuation coefficient of the internal material to X-rays.

[0011] Preferably, in step S3, an X-ray feature image is obtained after preprocessing and feature extraction. With ultrasonic feature spectrum The system integrates and jointly decides on the two information sources to achieve high-confidence classification and accurate positioning. This process is accomplished by a multi-channel deep learning fusion decision network, and its overall architecture and workflow are as follows: In the input layer and spatiotemporal registration, the first step is to prepare the input data. The network has two input channels, namely... and Before training and inference, the system performs spatial registration by pre-acquiring calibration samples at known locations to establish an affine transformation matrix from the ultrasonic scan line coordinate system to the X-ray image pixel coordinate system, and then... Resampling to Strict alignment is achieved in the spatial dimension; for time series data, timestamp synchronization is ensured by encoder trigger signals. In the dual-branch deep feature extraction network, the network adopts a parallel dual-branch encoder structure, which is optimized for the characteristics of the two modalities of data respectively: X-ray image branch: This branch is... For input, a convolutional neural network with ResNet50 as the backbone is used. The network extracts hierarchical features from low-level edge texture to high-level semantics through a series of convolutional layers, batch normalization layers, and activation functions (such as ReLU). To preserve spatial localization information, we extract the feature maps before downsampling in the last three layers of the backbone network, denoted as set. It has high spatial resolution; Ultrasound spectrum branch: This branch is based on As the input, since its data format is similar to that of an image but its physical meaning is different, a relatively lightweight CNN (e.g., 4-6 convolutional blocks) is used as the feature extractor; this network focuses on learning the patterns related to interface reflection and scattering in ultrasonic features, and at the same time, extracts its multi-scale feature maps, denoted as set. ; In the adaptive feature fusion based on the attention mechanism, firstly, the feature maps of the same scale from the two branches are subjected to 1x1 convolution to unify the number of channels, and then a cross-modal attention weight map is calculated. The specific calculation method is as follows: ; in, This is represented as an attention weight map for the X-ray features at the i-th position. Represented as Convolution operation, Let represent the X-ray feature at the i-th position. This is represented as performing a global average pooling operation on the ultrasonic features to obtain their global context information; This represents the ultrasonic feature at the i-th position; This weight graph This indicates the importance of X-ray features at each location in space. Similarly, an attention weight map for ultrasound features can be calculated. ; Ultimately, the fusion features at this scale It is obtained by weighted sum of the features of the two modalities and their respective attention weights.

[0012] Preferably, in step S4, after the intelligent decision network completes the analysis and calculation, the system enters the execution phase. The data processing and intelligent decision module generates a unique identity data packet containing a timestamp or location index for each potato that has completed the inspection. The core content of this data packet includes two parts: first, the final binary classification label output by the network ("qualified product" or "product with stone defects"), represented by a decision code; second, the detailed positioning information corresponding to the label. If it is a defective product, the precise coordinate data of its internal stone region is added. For example, in pixel-level segmentation, it is represented as a set of polygon vertex coordinates of one or more contours, or in three-dimensional region positioning, it is represented as the center point coordinates and size of a bounding box. This data packet is transmitted in real time and with low latency to the main controller of the sorting actuator via a high-speed industrial fieldbus. At the same time, the potatoes on the conveyor line are moving at a constant speed. A high-speed tracking algorithm runs in the main controller. Based on the real-time conveyor belt position fed back by the encoder, it dynamically tracks the movement trajectory of each potato that has been decided from the detection point to the sorting spray point. It also converts the static stone positioning coordinates stored in the data packet into the dynamic target position of the potato in the nozzle array coordinate system at the time of sorting, according to the speed and direction of movement. When a potato deemed defective arrives directly below the sorting port, a trigger signal is instantly generated. The core of the sorting actuator is an array-type jetting unit composed of multiple independently controllable high-speed solenoid valves. The controller precisely selects and activates a set of corresponding solenoid valves based on the real-time calculated dynamic target position. The high-pressure gas in the compressed air pipeline is released within milliseconds, forming a set of precisely focused high-intensity air jets. This jet acts vertically on a calculated specific position on the potato surface, applying a precise lateral impact force, causing its trajectory to deflect, thus detaching it from the main conveyor line and causing it to fall into the defective product collection channel or container.

[0013] The technical effects and advantages of this invention are as follows: High detection accuracy: X-rays are sensitive to high-density foreign objects, while ultrasound is sensitive to the acoustic impedance difference at solid-solid interfaces (the interface between potato tissue and stones). The combination of these two technologies, with their complementary advantages, significantly improves the recognition rate and positioning accuracy of embedded stones in complex backgrounds (such as potato variety, size, and internal physiological changes), while greatly reducing the false detection rate and missed detection rate. Intelligent decision-making: The end-to-end feature fusion and decision-making are performed using a deep learning model, which avoids the subjectivity and limitations of manually designed features in traditional methods. The model has self-learning and generalization capabilities and can adapt to potatoes from different origins and varieties. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0015] Figure 2 This is a schematic diagram of the X-ray image preprocessing workflow of the present invention.

[0016] Figure 3 This is a schematic diagram of the deep learning decision-making process of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 As shown, this invention provides a method for detecting embedded stones in potatoes using multimodal sensing and intelligent decision-making, comprising: S1: The potato to be tested passes through the detection area in a single row, and the X-ray imaging unit and the high-frequency ultrasonic scanning unit are triggered simultaneously to acquire its X-ray penetration image and ultrasonic echo signal, respectively.

[0019] In step S1, the variable frequency speed-regulating conveyor belt is activated, and the potatoes to be tested enter the detection area in a single, uniform row through the vibration sorting device and spacing limiting plate at the entrance. Within the detection area, an X-ray imaging unit and a high-frequency ultrasonic scanning unit are deployed in parallel. When a potato reaches the starting point of the detection area, multiple photoelectric encoders detect the leading edge position in real time and immediately generate a synchronous trigger pulse signal. This pulse signal is received by the central synchronous controller and simultaneously drives two sensing units. For the X-ray imaging unit, the controller triggers the instantaneous emission of a low-energy, short-pulse fan-shaped X-ray source. The X-rays that penetrate the potato are captured by the linear array detector on the opposite side, generating a grayscale projection image that reflects the two-dimensional density distribution and structure inside the potato in a single exposure. This image is sensitive to the attenuation difference between pebbles (high atomic number, high density) and potato tissue (mainly water and organic matter, low density). For the high-frequency ultrasonic scanning unit, the controller activates multiple high-frequency ultrasonic transducers within the same millisecond time window. The unit operates in pulse-echo mode: the transducer first emits a focused short ultrasonic pulse onto the potato surface, and then quickly switches to receiving mode. When the ultrasonic waves propagate inside and encounter the acoustic impedance interface between the potato tissue and the embedded stones, reflected echoes are generated. These raw A-scan (amplitude-time) echo signals carrying depth and interface characteristic information are synchronously recorded by a high-speed data acquisition card. To improve spatial resolution, the transducer can perform a transverse mechanical scan as the potato passes through, thereby acquiring acoustic data from multiple internal profiles.

[0020] S2: Denoise and contrast enhancement are performed on X-ray images; time-frequency analysis is performed on ultrasonic echo signals to extract time-domain and frequency-domain feature maps reflecting internal structural discontinuities.

[0021] In step S2, after acquiring the synchronously collected raw multimodal data, the system executes a preprocessing process in parallel to transform the raw multimodal data into standardized feature representations. In X-ray image preprocessing, the raw images received by the X-ray detector typically contain system noise, scattering noise, and background fluctuations; the following steps are used for processing: Background correction and noise reduction: First, perform dark field correction: from the original image Subtracting the dark field image without X-rays To eliminate detector dark current noise; Gain normalization is then performed using a standard reference image. Perform division operations to correct the non-uniformity of the detector pixel response; Corrected image The specific calculation method is as follows: ; Finally, an adaptive nonlocal mean filtering denoising algorithm is applied to effectively filter out random noise while preserving the edge details of the stones and tissues. In contrast and detail enhancement, to highlight the contrast between the stones (usually manifested as local bright areas) and the surrounding tissue, a contrast-limited adaptive histogram equalization algorithm is adopted. This algorithm divides the image into several sub-regions, performs histogram equalization on each, and prevents excessive amplification of noise by limiting contrast, effectively enhancing subtle density difference information.

[0022] Finally, the preprocessed X-ray feature image is obtained. Its pixel value directly reflects the relative attenuation coefficient of the internal material to X-rays; In ultrasonic echo signal processing and feature extraction, the original ultrasonic A-scan signal The signal-to-noise ratio is low, and the information about the stone interface is submerged in the complex tissue structure echoes; the following is an in-depth analysis from the two dimensions of time domain and time-frequency domain. In the bandpass filtering of signal preprocessing, a bandpass filter matching the transducer's center frequency is designed to filter out low-frequency oscillations and high-frequency electronic noise; the specific method for calculating the filtered signal is as follows: ; in, This is expressed as the frequency response of a bandpass filter; To compensate for the exponential attenuation of ultrasound waves propagating in materials, a gain function that increases exponentially with time is applied, namely: ,in, Represented as a time-varying gain function, Represented as the gain growth factor; In time-frequency analysis and feature map generation, the preprocessed signal The changes in frequency components cannot be simultaneously reflected at a single point in time; therefore, continuous wavelet transform is further used for time-frequency analysis, and the specific calculation method is as follows: ; in, This is represented as the result of continuous wavelet transform, where 'a' represents the scaling factor. Represented as the translation factor, Represented as a sub-wavelet; The main characteristics of stones under ultrasound are instantaneous high-energy reflected echoes and abrupt changes in spectral characteristics caused by irregular scattering at the interface. Therefore, two key feature maps are extracted from the CWT coefficient matrix, including the instantaneous energy envelope map and the instantaneous center frequency map. In the instantaneous energy envelope map, the time location of the strongly reflective interface is highlighted by integrating the squared magnitudes of the CWT coefficients at all scales corresponding to each time point t. The specific calculation method is as follows: ; in, Represented as an instantaneous energy envelope diagram; The instantaneous center frequency diagram reflects the change of the signal's dominant frequency over time; reflection from the stone interface may cause this. The sudden shift in frequency response is specifically calculated as follows: ; in, Represented as Representation mapped to the frequency domain; Ultimately, and By arranging and combining elements along the potato scanning direction, a two-dimensional ultrasonic feature map is generated, where pixel values ​​represent different physical meanings (energy, center frequency). This atlas visually reveals the structural discontinuities and acoustic heterogeneity within the potato along the scanning path.

[0023] Thus, the original multimodal sensing data has been transformed into two highly condensed feature images / maps aligned in space (X-ray image) and scan path-time frequency (ultrasound map), providing standardized, high-quality input for the next stage of intelligent fusion decision-making.

[0024] S3: The preprocessed X-ray image and ultrasonic feature map are spatiotemporally registered and input into a trained multi-channel deep learning fusion decision network. This network extracts deep features from the two modalities in parallel, performs adaptive weighted fusion at the feature layer, outputs the classification probability of whether each potato contains embedded stones, and performs pixel-level localization of the stone positions.

[0025] In step S3, X-ray feature images are obtained after preprocessing and feature extraction. With ultrasonic feature spectrum The system integrates and jointly decides on the two information sources to achieve high-confidence classification and accurate positioning. This process is accomplished by a multi-channel deep learning fusion decision network, and its overall architecture and workflow are as follows: In the input layer and spatiotemporal registration, the first step is to prepare the input data. The network has two input channels, namely... and Before training and inference, the system performs spatial registration by pre-acquiring calibration samples at known locations to establish an affine transformation matrix from the ultrasonic scan line coordinate system to the X-ray image pixel coordinate system, and then... Resampling to Strict alignment is achieved in the spatial dimension; for time series data, timestamp synchronization is ensured by encoder trigger signals. In the dual-branch deep feature extraction network, the network adopts a parallel dual-branch encoder structure, which is optimized for the characteristics of the two modalities of data respectively: X-ray image branch: This branch is... For input, a convolutional neural network with ResNet50 as the backbone is used. The network extracts hierarchical features from low-level edge texture to high-level semantics through a series of convolutional layers, batch normalization layers, and activation functions (such as ReLU). To preserve spatial localization information, we extract the feature maps before downsampling in the last three layers of the backbone network, denoted as set. It has high spatial resolution; Ultrasound spectrum branch: This branch is based on As the input, since its data format is similar to that of an image but its physical meaning is different, a relatively lightweight CNN (e.g., 4-6 convolutional blocks) is used as the feature extractor; this network focuses on learning the patterns related to interface reflection and scattering in ultrasonic features, and at the same time, extracts its multi-scale feature maps, denoted as set. ; In the adaptive feature fusion based on the attention mechanism, firstly, the feature maps of the same scale from the two branches are subjected to 1x1 convolution to unify the number of channels, and then a cross-modal attention weight map is calculated. The specific calculation method is as follows: ; in, This is represented as an attention weight map for the X-ray features at the i-th position. Represented as Convolution operation, Let represent the X-ray feature at the i-th position. This is represented as performing a global average pooling operation on the ultrasonic features to obtain their global context information; This represents the ultrasonic feature at the i-th position; This weight graph This indicates the importance of X-ray features at each location in space. Similarly, an attention weight map for ultrasound features can be calculated. ; Ultimately, the fusion features at this scale It is obtained by weighting the features of the two modalities and their respective attention weights. The specific calculation method is as follows: ; in, This is represented as element-wise multiplication; The multi-channel deep learning fusion decision network can rely more on X-ray density information in regions with uniform internal structure of potatoes, while assigning higher weights to ultrasonic reflection features in regions suspected of having sharp interfaces. In multi-task decoding and output, the fused multi-scale features They are fed into a dual-head decoder to perform classification and localization tasks respectively: In the existence determination of the classification head, the highest level of fused features are used. After global average pooling, the input is a small classifier consisting of a fully connected layer and a softmax function, and the output is a two-dimensional probability vector. ,in, This represents the probability that a potato does not contain any embedded stones. This represents the probability that the potato contains embedded stones. A threshold value is set to make the final classification decision; that is, if... If the value is greater than the threshold, the potato is considered to contain embedded stones; otherwise, the potato is considered not to contain embedded stones. In the localization of the stone region segmentation, a U-Net-like decoder structure is adopted to fuse features at different scales. By fusing the feature map with the corresponding resolution in the decoder through skip connections, and after a series of upsampling and convolution operations, the final output is a map that is identical to the input. A probability heatmap with the same spatial dimensions; the value of each pixel on the heatmap represents the probability that the location belongs to the pebble region. By setting a threshold and performing connected component analysis, the precise pixel-level outline of the pebble region can be obtained. The intelligent decision-making network not only outputs high-confidence classification results (containing stones / not containing stones), but also provides refined positioning information, providing precise coordinate instructions for subsequent location-based sorting actions, thus realizing a complete intelligent closed loop from perception and understanding to decision-making.

[0026] S4: The judgment results and positioning information output by the intelligent decision network are sent to the sorting execution mechanism in real time to remove potatoes containing embedded stones from the production line.

[0027] In step S4, after the intelligent decision network completes the analysis and calculation, the system enters the execution phase. The data processing and intelligent decision module generates a unique identity data packet for each potato that has completed the inspection, containing a timestamp or location index. The core content of this data packet includes two parts: first, the final binary classification label output by the network ("qualified product" or "product with stone defects"), represented by a decision code; second, the detailed location information corresponding to the label. If it is a defective product, the precise coordinate data of its internal stone area is added. For example, in pixel-level segmentation, it is represented as a set of polygon vertex coordinates of one or more contours, or in three-dimensional region positioning, it is represented as the center point coordinates and size of a bounding box. This data packet is transmitted in real time and with low latency to the main controller of the sorting actuator via a high-speed industrial fieldbus. At the same time, the potatoes on the conveyor line are moving at a constant speed. A high-speed tracking algorithm runs in the main controller. Based on the real-time conveyor belt position fed back by the encoder, it dynamically tracks the movement trajectory of each potato that has been decided from the detection point to the sorting spray point. It also converts the static stone positioning coordinates stored in the data packet into the dynamic target position of the potato in the nozzle array coordinate system at the time of sorting, according to the speed and direction of movement. When a potato deemed defective arrives directly below the sorting port, a trigger signal is instantly generated. The core of the sorting actuator is an array-type jetting unit composed of multiple independently controllable high-speed solenoid valves. The controller precisely selects and activates a set of corresponding solenoid valves based on the real-time calculated dynamic target position. The high-pressure gas in the compressed air pipeline is released within milliseconds, forming a set of precisely focused high-intensity air jets. This jet acts vertically on a specific calculated position on the potato surface, applying a precise lateral impact force, causing its trajectory to deflect, thus detaching it from the main conveyor line and causing it to fall into the defective product collection channel or container. The entire process strictly adheres to the principle of real-time operation; qualified products continue to move smoothly along the main conveyor line without any interference, entering the next process; the system's human-machine interface updates and accumulates statistical information such as sorting quantity and defect rate in real time, and issues an alarm when an abnormality occurs in the actuator or when the decision confidence is too low.

[0028] This invention also provides a potato embedded stone detection system with multimodal sensing and intelligent decision-making, comprising: Multimodal data synchronous acquisition module: Enables the potato to be tested to pass through the detection area in a single row, synchronously triggering the X-ray imaging unit and the high-frequency ultrasonic scanning unit to acquire its X-ray penetration image and ultrasonic echo signal respectively; Multimodal data preprocessing and feature extraction module: performs noise reduction and contrast enhancement on X-ray images; performs time-frequency analysis on ultrasonic echo signals to extract time-domain and frequency-domain feature maps reflecting internal structural discontinuities; Multimodal feature fusion and intelligent decision-making module: The preprocessed X-ray image and ultrasonic feature map are spatiotemporally registered and input into a trained multi-channel deep learning fusion decision network; the network extracts deep features of the two modalities in parallel, performs adaptive weighted fusion at the feature layer, outputs the classification probability of whether each potato has embedded stones, and performs pixel-level localization of the stone positions; The sorting module executes the sorting action by sending the judgment results and location information output by the intelligent decision network to the sorting execution mechanism in real time, thereby removing potatoes containing embedded stones from the production line.

[0029] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting embedded stones in potatoes using multimodal sensing and intelligent decision-making, characterized in that, include: S1: The potato to be tested passes through the detection area in a single row, and the X-ray imaging unit and the high-frequency ultrasonic scanning unit are triggered simultaneously to acquire its X-ray penetration image and ultrasonic echo signal, respectively. S2: Denoise and contrast enhancement of X-ray images; Time-frequency analysis of ultrasonic echo signals to extract time-domain and frequency-domain feature maps reflecting internal structural discontinuities; S3: The preprocessed X-ray image and the ultrasonic feature map are spatiotemporally registered and input into the trained multi-channel deep learning fusion decision network. The network extracts deep features of the two modalities in parallel, performs adaptive weighted fusion at the feature layer, outputs the classification probability of whether each potato has embedded stones, and performs pixel-level localization of the stone positions. S4: The judgment results and positioning information output by the intelligent decision network are sent to the sorting execution mechanism in real time to remove potatoes containing embedded stones from the production line.

2. The method for detecting embedded stones in potatoes using multimodal sensing and intelligent decision-making according to claim 1, characterized in that: In step S1, the variable frequency speed control conveyor belt is started, and the potatoes to be tested are arranged in a single row and uniformly placed into the detection area through the vibration sorting device and the spacing limit plate at the entrance. In the detection area, X-ray imaging unit and high-frequency ultrasonic scanning unit are deployed in parallel. When the potato arrives at the starting point of the detection area, multiple photoelectric encoders detect the leading edge position in real time and immediately generate a synchronous trigger pulse signal. After the pulse signal is received by the central synchronous controller, it will drive two sensing units at the same time.

3. The method for detecting embedded stones in potatoes using multimodal sensing and intelligent decision-making according to claim 1, characterized in that: In S2, after acquiring the synchronously collected raw multimodal data, the system executes the preprocessing process in parallel to transform the raw multimodal data into a standardized feature representation; in the X-ray image preprocessing, the raw image received by the X-ray detector contains system noise, scattering noise and background fluctuations; The following steps are used for processing: Background correction and noise reduction: First, perform dark field correction: from the original image Subtracting the dark field image without X-rays To eliminate detector dark current noise; Gain normalization is then performed using a standard reference image. Perform division operations to correct the non-uniformity of the detector pixel response; Corrected image The specific calculation method is as follows: ； in, It is represented as a dark field image.

4. The method for detecting embedded stones in potatoes using multimodal sensing and intelligent decision-making according to claim 1, characterized in that: In ultrasonic echo signal processing and feature extraction, the original ultrasonic A-scan signal The signal-to-noise ratio is low, and the information of the stone interface is submerged in the complex tissue structure echo. In-depth analysis is carried out from both time domain and time-frequency domain dimensions. In the bandpass filtering of signal preprocessing, a bandpass filter matching the transducer's center frequency is designed to filter out low-frequency oscillations and high-frequency electronic noise; the specific method for calculating the filtered signal is as follows: ； in, This is expressed as the frequency response of a bandpass filter; To compensate for the exponential attenuation of ultrasound waves propagating in materials, a gain function that increases exponentially with time is applied, namely: ,in, Represented as a time-varying gain function, This is expressed as the gain growth factor; In time-frequency analysis and feature map generation, the preprocessed signal The changes in frequency components cannot be simultaneously reflected at a single point in time; therefore, continuous wavelet transform is further used for time-frequency analysis, and the specific calculation method is as follows: ； in, This is represented as the result of continuous wavelet transform, where 'a' represents the scaling factor. Represented as the translation factor, It is represented as a sub-wavelet.

5. The method for detecting embedded stones in potatoes using multimodal sensing and intelligent decision-making according to claim 1, characterized in that: In the instantaneous energy envelope map, the time location of the strongly reflective interface is highlighted by integrating the squared magnitudes of the CWT coefficients at all scales corresponding to each time point t. The specific calculation method is as follows: ； in, Represented as an instantaneous energy envelope diagram; The instantaneous center frequency diagram reflects the change of the signal's dominant frequency over time. The specific method for calculating the instantaneous center frequency diagram is as follows: ； in, Represented as Representation mapped to the frequency domain; Ultimately, and An ultrasonic feature map is generated by arranging and combining the data along the potato scanning direction. .

6. The method for detecting embedded stones in potatoes using multimodal sensing and intelligent decision-making according to claim 1, characterized in that: In step S3, X-ray feature images are obtained after preprocessing and feature extraction. With ultrasonic feature map The system integrates and jointly decides on the two information sources to achieve high-confidence classification and accurate positioning. This process is accomplished by a multi-channel deep learning fusion decision network, and its overall architecture and workflow are as follows: In the input layer and spatiotemporal registration, the first step is to prepare the input data. The network has two input channels, namely... and Before training and inference, the system performs spatial registration by pre-acquiring calibration samples at known locations to establish an affine transformation matrix from the ultrasonic scan line coordinate system to the X-ray image pixel coordinate system, and then... Resampling to Strict alignment is achieved in the spatial dimension; for time-series data, timestamp synchronization is ensured by encoder trigger signals.

7. The method for detecting embedded stones in potatoes using multimodal sensing and intelligent decision-making according to claim 6, characterized in that: In the dual-branch deep feature extraction network, the network adopts a parallel dual-branch encoder structure, which is optimized for the characteristics of the two modalities of data respectively: X-ray image branch: This branch is... For input, a convolutional neural network with ResNet50 as the backbone is used; the network extracts hierarchical features from low-level edge texture to high-level semantics through a series of convolutional layers, batch normalization layers, and activation functions; the feature maps before downsampling of the last three layers of the backbone network are extracted and denoted as set. ; Ultrasound spectrum branch: This branch is based on For the input, a relatively lightweight CNN is used as the feature extractor, and its multi-scale feature maps are extracted, denoted as set. ; In attention-based adaptive feature fusion, firstly, the same-scale feature maps from the two branches are subjected to 1x1 convolution to unify the number of channels. Then, a cross-modal attention weight map is calculated. Finally, the fused features at this scale are... It is obtained by weighted sum of the features of the two modalities and their respective attention weights.

8. The method for detecting embedded stones in potatoes using multimodal sensing and intelligent decision-making according to claim 6, characterized in that: In multi-task decoding and output, the fused multi-scale features They are fed into a dual-head decoder to perform classification and localization tasks respectively; In the existence determination of the classification head, the highest level of fused features are used. After global average pooling, the input is a small classifier consisting of a fully connected layer and a softmax function, and the output is a two-dimensional probability vector. ,in, This represents the probability that a potato does not contain any embedded stones. This represents the probability that the potato contains embedded stones; a threshold value is set to make the final classification decision; that is, if... If the value is greater than the threshold, the potato is considered to contain embedded stones; otherwise, the potato is considered not to contain embedded stones. In the localization of the stone region segmentation, a U-Net-like decoder structure is adopted to fuse features at different scales. By fusing the feature map with the corresponding resolution in the decoder through skip connections, and after a series of upsampling and convolution operations, the final output is a map that is identical to the input. Probability heatmaps of spaces with the same dimensions.

9. The method for detecting embedded stones in potatoes using multimodal sensing and intelligent decision-making according to claim 1, characterized in that: In step S4, after the intelligent decision network completes the analysis and calculation, the system enters the execution phase. The data processing and intelligent decision module generates an identity data packet for each potato that has completed the inspection. The data packet includes two parts: one is the final binary classification label output by the network, represented by a decision code; the other is the detailed location information corresponding to the label. If it is a defective product, the precise coordinate data of its internal stone area is added. The data packet is transmitted to the main controller of the sorting actuator. The potatoes on the conveyor line are moving at a constant speed. The main controller runs a high-speed tracking algorithm. Based on the real-time conveyor belt position fed back by the encoder, it dynamically tracks the movement trajectory of each potato that has been decided from the detection point to the sorting spray point. It also converts the static stone positioning coordinates stored in the data packet into the dynamic target position of the potato in the nozzle array coordinate system at the time of sorting based on the movement speed and direction.

10. A potato embedded stone detection system based on multimodal sensing and intelligent decision-making, using the potato embedded stone detection method based on multimodal sensing and intelligent decision-making as described in any one of claims 1-9, characterized in that: Multimodal data synchronous acquisition module: Enables the potato to be tested to pass through the detection area in a single row, synchronously triggering the X-ray imaging unit and the high-frequency ultrasonic scanning unit to acquire its X-ray penetration image and ultrasonic echo signal respectively; Multimodal data preprocessing and feature extraction module: performs noise reduction and contrast enhancement on X-ray images; performs time-frequency analysis on ultrasonic echo signals to extract time-domain and frequency-domain feature maps reflecting internal structural discontinuities; Multimodal feature fusion and intelligent decision-making module: The preprocessed X-ray image and ultrasonic feature map are spatiotemporally registered and input into a trained multi-channel deep learning fusion decision network; the network extracts deep features of the two modalities in parallel, performs adaptive weighted fusion at the feature layer, outputs the classification probability of whether each potato has embedded stones, and performs pixel-level localization of the stone positions; The sorting module executes the sorting action by sending the judgment results and location information output by the intelligent decision network to the sorting execution mechanism in real time, thereby removing potatoes containing embedded stones from the production line.

Images