Multimodal shape perception based dopant automatic rejection method and device and computer readable storage medium

Abstract

The application provides a kind of based on multimodal shape perception's adulteration automatic rejection method, device and computer readable storage medium, belong to computer vision and wisdom agricultural technology field.Method includes:S10 obtains material image;S20 is preprocessed to the material image, obtains preprocessed image;S30 is based on the automatic rejection model of training, input preprocessed image, generates automatic rejection execution signal, executes automatic rejection.The application provides a kind of based on multimodal shape perception's adulteration automatic rejection method first based on the multimodal feature fusion network constructed, strengthens the shape feature of adulteration in material image, then based on shape detection model, anchor the information and coordinates of adulteration from product, method simultaneously fuses the color information and shape information of adulteration, improves the sorting precision when adulteration and product color are highly similar, reduces the probability of system false detection, missed detection.

Description

Technical Field

[0001] This application belongs to the field of computer vision and smart agriculture technology, and specifically relates to a method, apparatus and computer-readable storage medium for automatic removal of dopants based on multimodal shape perception. Background Technology

[0002] In the refining and processing of agricultural products such as tea and vegetables, removing impurities from the materials, such as removing green stems from tea products and straw from dried vegetable products, is a key process to ensure product quality.

[0003] Traditional sorting methods rely on manual labor, which is inefficient, costly, and the sorting accuracy is greatly affected by human experience. Existing automated sorting technologies mostly use color sorters, which rely solely on color characteristics for sorting. When adulterants are very similar in color to the product, false detections and missed detections are likely to occur, resulting in a significant decrease in sorting accuracy.

[0004] Therefore, there is an urgent need to develop an automated dopant removal method that is highly accurate, has low latency, and is widely adaptable. Summary of the Invention

[0005] Based on the aforementioned technical needs, this application provides an automatic dopant removal method based on multimodal shape perception to solve the technical problem in the prior art where the dopant and product colors are highly similar when using a color sorter to remove dopant, resulting in low sorting accuracy.

[0006] To achieve the above objectives, the technical solution of this application is as follows: This application provides an automatic dopant removal method based on multimodal shape awareness, comprising the following steps: S10 Acquire material image; S20 preprocesses the material image to obtain a preprocessed image; S30 is a trained automatic elimination model that takes a pre-processed image as input and generates an automatic elimination execution signal. The automatic elimination model includes the following process: S31 Extract multimodal features from the preprocessed image; S32 Based on the shape detection model, the multimodal features are processed to obtain the category confidence of the classification category; S33 Based on the comparison result between the category confidence score and the preset category confidence score threshold, an automatic rejection execution signal is generated, wherein the automatic rejection execution signal contains the coordinate information of the dopant; Preferably, "the YOLO26-OBB is used to extract deep semantic features of an image and obtain a global feature map" includes: processing the preprocessed image based on a low-dimensional feature fusion mechanism to obtain a dimensionality-reduced global feature map.

[0007] Preferably, "the Strip R-CNN is used to extract predetermined local features and obtain local feature maps" includes: obtaining the dimensionality-reduced global feature map output by YOLO26-OBB, and extracting candidate morphological feature maps using a multi-scale sampling network; the morphological feature maps are used to describe the morphological features of the dopant. If the morphological features meet the preset dopant morphology conditions, the morphological features of the dopant are identified as candidate features, and the candidate positions are determined through a region proposal network to suppress redundant features and output a local feature map based on the first candidate features.

[0008] Preferably, "the Mamba-YOLO used to fuse the global feature map and the local feature map to obtain the multimodal features" includes: performing feature normalization and feature dimension alignment processing on the global feature map output by YOLO26-OBB and the local feature map output by Strip R-CNN to obtain a normalized global feature map and a local feature map. Normalization and feature dimension alignment are performed to associate and model the global and local feature maps. In the spatial dimension, the second candidate feature is determined based on the material distribution characteristics and dopant morphology feature map, and invalid features are suppressed. The attention-based feature weighted concatenation outputs a multimodal fusion feature map based on the first and second candidate features.

[0009] Preferably, the shape detection model is selected from at least one of a first shape detection model, a second shape detection model, and a third shape detection model; The first shape detection model includes an elliptic parametric regression model; the second shape detection model includes a key point contour fitting model; and the third shape detection model includes a Fourier series distance field shape perception model.

[0010] Preferably, the shape detection model simultaneously runs the first shape detection model, the second shape detection model, and the third shape detection model.

[0011] Preferably, the process of training the automatic elimination model includes a first-stage training and a second-stage training. The first-stage training is pre-trained on the COCO-OBB rotating object detection dataset, and the second-stage training is fine-tuned on the detection dataset, freezing the backbone network and training only the shape detection model and the Mamba-YOLO fusion model.

[0012] Preferably, in step S20, "preprocessing the material image to obtain a preprocessed image" includes at least one of distortion correction processing, white balance processing, and data enhancement processing on the material image.

[0013] This application provides an automatic dopant removal device based on multimodal shape sensing, comprising: The material image acquisition module is used to acquire material images; The image preprocessing module is used to preprocess the material image to obtain a preprocessed image; A multimodal feature extraction module is used to extract multimodal features from the preprocessed image; The shape detection module is used to process the multimodal features based on the shape detection model to obtain the category confidence of the classification category; The confidence level judgment module is used to generate an automatic rejection execution signal based on the comparison result between the category confidence level and the preset category confidence level threshold. The automatic rejection execution signal contains the coordinate information of the dopant. The step of "extracting multimodal features from the preprocessed image" includes: Based on the constructed multimodal feature fusion network, multimodal features are extracted from the preprocessed image. The multimodal feature fusion network adopts a fusion architecture of YOLO26-OBB, Strip R-CNN and Mamba-YOLO. The YOLO26-OBB is used to extract deep semantic features of images and obtain a global feature map. The Strip R-CNN is used to extract predetermined local features and obtain local feature maps; The Mamba-YOLO is used to fuse the global feature map and the local feature map to obtain the multimodal features.

[0014] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the aforementioned multimodal shape-aware automatic dopant removal method.

[0015] Compared with the prior art, this application has at least the following beneficial effects: The technical solution provided in this application includes the following steps: S10 acquiring a material image; S20 preprocessing the material image to obtain a preprocessed image; S30 based on a trained automatic rejection model, inputting the preprocessed image to generate an automatic rejection execution signal. First, based on the constructed multimodal feature fusion network, the shape features of dopants in the material image are enhanced. Then, based on the shape detection model, the information and coordinates of dopants are anchored from the product. The method simultaneously integrates the color information and shape information of dopants, improving the sorting accuracy when the color of dopants is highly similar to that of the product, and reducing the probability of false detection and missed detection in the system. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the network structure for an embodiment of the present application that proposes an automatic dopant removal method based on multimodal shape awareness.

[0017] Figure 2 This is a schematic diagram of an automatic dopant removal method based on multimodal shape perception according to an embodiment of this application; Figure 3 This is another schematic diagram of the automatic dopant removal method based on multimodal shape perception according to an embodiment of this application; Figure 4 This is a schematic diagram of an automated dopant device based on multimodal shape sensing, according to an embodiment of this application. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The technical solutions of this application will be further described below with reference to the accompanying drawings of the embodiments, and this application is not limited to the following specific implementation methods.

[0019] It should be understood that the same or similar reference numerals in the accompanying drawings of the embodiments correspond to the same or similar components. In the description of this application, it should be understood that if terms such as "upper," "lower," "inner," "outer," "left," "right," "front," "rear," "top," and "bottom" indicate directions or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, they are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the structure or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms describing positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limitations on this patent. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0020] The following is combined Figure 1 The present application will be further described in detail with reference to specific embodiments.

[0021] In one specific embodiment of this application, please refer to Figure 1 This application provides an automatic dopant removal method based on multimodal shape perception, comprising the following steps: S10 Acquire material images; in some specific embodiments, material images are acquired based on industrial cameras mounted on a material conveying device, such as a conveyor belt. For example, the acquired material images have a resolution of 4K and a frame rate of 30fps.

[0022] S20 preprocesses the material image to obtain a preprocessed image; firstly, the material image is preprocessed to obtain a preprocessed image. In some recommended embodiments, preprocessing the material image is not limited to distortion correction and white balance processing to eliminate the effects of industrial camera distortion and uneven lighting. A shape-aware data augmentation strategy is adopted, including geometric transformation, shape enhancement, and color enhancement, to expand the dataset size, improve the model's generalization ability, and adapt to the posture, shape, and lighting changes of materials in actual industrial scenarios.

[0023] S30 is a trained automatic elimination model that takes a pre-processed image as input and generates an automatic elimination execution signal. The automatic elimination model includes the following process: S31 Extracting multimodal features from the preprocessed image; wherein, "extracting multimodal features from the preprocessed image" includes: Based on the constructed multimodal feature fusion network, multimodal features are extracted from the preprocessed image. The multimodal feature fusion network adopts a fusion architecture of YOLO26-OBB, Strip R-CNN, and Mamba-YOLO. YOLO26-OBB is used to extract deep semantic features of the image to obtain a global feature map; Strip R-CNN is used to extract predetermined local features to obtain a local feature map; and Mamba-YOLO is used to fuse the global feature map and the local feature map to obtain the multimodal features.

[0024] S32 Based on the shape detection model, the multimodal features are processed to obtain the category confidence of the classification category; In some feasible implementations, the shape detection model is selected from at least one of a first shape detection model, a second shape detection model, and a third shape detection model. The first shape detection model includes an elliptic parametric regression model; the second shape detection model includes a keypoint contour fitting model; and the third shape detection model includes a Fourier series distance field shape perception model. The first shape detection model, the second shape detection model, and the third shape detection model can be used individually, or, to improve shape detection efficiency and effectiveness, they can be used simultaneously.

[0025] Furthermore, the elliptic parameter regression model outputs an elliptic parameter vector of the dopant, accurately capturing the geometric features of dopants with regular elliptical shapes. Specifically, the fused high-quality multimodal features are input into the dedicated convolutional branch of the elliptic parameter regression model. After multiple 1×1 convolutions, feature dimensionality reduction and feature filtering are completed, focusing on the target morphological features and eliminating irrelevant interference features. Subsequently, the effective features are linearly mapped through a fully connected layer, and the end-to-end regression outputs a five-dimensional elliptic parameter vector, which consists of the dopant center coordinates (x, y), major axis length, minor axis length, and rotation angle. Combined with the model's preset standard elliptical shape template for agricultural product dopant, the output parameters are normalized and corrected, and the similarity score between the actual target shape and the standard dopant elliptical shape is calculated. Based on the similarity score, the corresponding category confidence score is generated to accurately determine regular strip-shaped and elliptical dopants. At the same time, the output coordinate and angle parameters can be directly used for subsequent removal and localization, adapting to the detection scenario of slender dopants with regular shapes and postures.

[0026] The keypoint contour fitting model predicts key points on the dopant contour and fits elliptical parameters to adapt to dopant with irregular and slender shapes. Specifically, after multimodal features are input into the keypoint detection branch, a dedicated keypoint convolution prediction layer is used to densely predict multiple sets of preset keypoint coordinates, such as edge inflection points, upper and lower boundaries, and left and right endpoints of the dopant contour. For the massive number of predicted keypoints, a triple strategy of neighborhood denoising, distance filtering, and confidence filtering is used to remove invalid keypoints such as background noise, material edge interference, and abnormal offsets, retaining the effective keypoint set that fits the true contour of the dopant. The least squares method is used to iteratively fit the effective keypoints to reconstruct the complete contour curve of the dopant, eliminating contour distortion caused by local deformation and slight occlusion. Based on the fitted continuous contour curve, the geometric parameters such as the major axis, minor axis, rotation angle, and center coordinates of the optimal fitted ellipse are derived in reverse, and the contour shape matching confidence is calculated simultaneously to achieve accurate identification of non-standard slender dopant with curved, uneven thickness, and local irregularities, making up for the shortcoming of the elliptical parameter regression model, which only adapts to regular shapes.

[0027] The Fourier series distance field shape perception model achieves robust recognition of occluded and incomplete green stems by predicting the directed distance field and reconstructing the contour. Specifically, multimodal features are input into the distance field prediction branch, and the directed distance field value of the image is predicted pixel by pixel through a convolutional network. Based on the target contour, the model distinguishes between positive distance inside the target, zero distance at the contour boundary, and negative distance to the background, accurately delineating the boundary regions between dopants and normal materials and the background. For discrete distance field data caused by occlusion, breakage, and incompleteness, a multi-level Fourier series is used for continuous fitting, transforming discrete pixel distance values ​​into a continuous and smooth contour mathematical function, restoring the complete and realistic contour of dopants that are occluded, broken, or stacked. Based on the reconstructed complete contour, core shape features such as the target aspect ratio, contour area, curvature, and morphological compactness are accurately extracted and connected to the feature matching and scoring module. Similarity matching is performed with a standard dopant morphology feature library, and the final classification confidence and contour coordinate parameters are output. This effectively solves the problem of missed detection and false detection caused by dopant defects and occlusion in complex industrial scenarios, improving the robustness of the model.

[0028] In the above technical solution, firstly, based on the constructed multimodal feature fusion network, the shape features of dopants in the material image are enhanced. Then, based on the shape detection model, the information and coordinates of dopants are anchored from the product. The method simultaneously integrates the color information and shape information of the dopants, which improves the sorting accuracy when the dopants and the product colors are highly similar, and reduces the probability of false detection and missed detection in the system.

[0029] S33 Based on the comparison result between the category confidence score and the preset category confidence score threshold, an automatic rejection execution signal is generated, wherein the automatic rejection execution signal contains the coordinate information of the dopant; Based on a trained automatic removal model, a preprocessed image is input, and an automatic removal execution signal is generated. The automatic removal model includes the following processes: S31 Extracting multimodal features from the preprocessed image; wherein, "extracting multimodal features from the preprocessed image" includes: Based on the constructed multimodal feature fusion network, multimodal features are extracted from the preprocessed image. The multimodal feature fusion network adopts a fusion architecture of YOLO26-OBB, Strip R-CNN, and Mamba-YOLO. YOLO26-OBB is used to extract deep semantic features of the image to obtain a global feature map; Strip R-CNN is used to extract predetermined local features to obtain a local feature map; and Mamba-YOLO is used to fuse the global feature map and the local feature map to obtain the multimodal features.

[0030] Furthermore, multimodal features in the preprocessed image are first obtained based on a multimodal feature fusion network. The multimodal feature fusion network adopts a fusion architecture of YOLO26-OBB, Strip R-CNN, and Mamba-YOLO, wherein the YOLO26-OBB (e.g., OD Mamba) backbone is used to extract deep semantic features of the image. This backbone has linear complexity, which can improve inference speed while ensuring feature extraction accuracy.

[0031] "The YOLO26-OBB used to extract deep semantic features of an image and obtain a global feature map" includes: processing the preprocessed image based on a low-dimensional feature fusion mechanism to obtain a dimensionality-reduced global feature map.

[0032] First, the preprocessed images are uniformly scaled to the model's preset input size to complete image normalization and dimensionality adaptation. Then, the YOLO26-OBB backbone feature extraction network sequentially passes through shallow convolutional layers, a mid-layer feature stacking module, and a deep Mamba temporal modeling module. The shallow convolutional layers are responsible for extracting the basic texture, color, and edge features of the image. The mid-layer module completes the aggregation and dimensionality reduction of the basic features. The deep OD Mamba module relies on a linear state-space model to perform long-distance modeling of the global pixel correlation information of the image, capturing the overall distribution of materials, the global location of dopants, and the semantic information of the overall morphology. Finally, through the multi-scale feature output of the backbone network, global feature maps of different scales are generated, covering the global semantic features of large, medium, and small-sized dopants.

[0033] Specifically, YOLO26-OBB with the OD Mamba backbone is used to extract deep semantic features of images. This backbone abandons the quadratic complexity of traditional convolution operations and has linear operation characteristics, which can significantly improve the inference speed while ensuring the accuracy of feature extraction. The specific deep semantic extraction process is as follows: After the preprocessed standardized image is input into the YOLO26-OBB backbone network, it first extracts the basic texture, edge and color features of the image through the shallow C3k convolution module to complete the initial aggregation of low-dimensional features; the middle layer realizes multi-scale basic feature fusion, dimensionality reduction and spatial information enhancement by stacking the C3k2 module and the SPPF spatial pooling module, and retains the basic features of materials and dopants of different sizes.

[0034] Strip R-CNN extracts predetermined local features to obtain local feature maps. Strip R-CNN is an improved model based on Faster R-CNN, designed to improve detection speed and accuracy. It enhances feature extraction capabilities by using a densely connected feature pyramid network (Dilated Convolution). Strip R-CNN strengthens the feature capture capability of slender targets, adapting to the slender shapes of dopants.

[0035] like Figure 2 As shown, "the Strip R-CNN is used to extract predetermined local features and obtain local feature maps" includes: Step S210: Obtain the dimensionality-reduced global feature map output by YOLO26-OBB, and extract candidate morphological feature maps using a multi-scale sampling network; the morphological feature maps are used to describe the morphological features of the dopant; firstly, the basic shallow features output by YOLO26-OBB are received, and through a densely connected dilated convolutional feature pyramid network, convolutional kernels with different dilation rates are set to perform multi-scale refined feature sampling on the local area of ​​the image; firstly, the basic shallow features output by YOLO26-OBB are received, and through a densely connected dilated convolutional feature pyramid network, convolutional kernels with different dilation rates are set to perform multi-scale refined feature sampling on the local area of ​​the image; specifically, the model receives the shallow basic features output by YOLO26-OBB, discards the global invalid region features, and performs multi-scale sampling on the local refined region of the image through a densely connected multi-dilation rate dilated convolutional feature pyramid, covering the morphology of slender dopant of varying thickness and length.

[0036] Step S220: If the morphological features meet the preset dopant morphology conditions, the morphological features of the dopant are identified as candidate features, and the candidate positions are determined through the region proposal network to suppress redundant features and output a local feature map based on the first candidate features.

[0037] Subsequently, the image is filtered using a Region Proposal Network (RPN) to select candidate bounding boxes, focusing on local areas of suspected dopants that are thin, elongated, or irregular on the material surface, while filtering out large areas of background and invalid areas of normal material. Then, ROIAlign is used to precisely align the feature pixels of the candidate regions to avoid local feature shifts and information loss. Finally, feature enhancement extraction is performed on the filtered local regions to capture fine local features such as the thin edges of the dopants, local uneven textures, and narrow strip shapes, and output a high-precision local feature map.

[0038] By optimizing the Region Proposal Network (RPN) and combining it with a slender target anchor box mechanism, we can accurately screen strip-shaped local candidate regions of suspected dopants on the material surface and filter out large areas of normal material and background interference. Then, we use the ROIAlign pixel-level alignment method to correct the feature offset problem of candidate regions and retain the complete details of local regions. Finally, we use a local feature enhancement convolutional layer to focus on enhancing the fine local features such as the slender edges, uneven textures, narrow strip structures, and local breaks of dopants, while suppressing the smooth and regular features of normal materials. The final output is a high-precision and highly focused local feature map, which makes up for the lack of global features in capturing subtle local morphological and slender structural features.

[0039] The Mamba-YOLO feature fusion layer enables efficient fusion of global and local features, reducing background noise interference and improving feature utilization.

[0040] like Figure 3 As shown, "the Mamba-YOLO used to fuse the global feature map and the local feature map to obtain the multimodal features" includes: Step S310: Perform feature normalization and feature dimension alignment on the global feature map output by YOLO26-OBB and the local feature map output by Strip R-CNN to obtain normalized global feature map and local feature map; First, perform dimension unification and normalization on the global feature map output by YOLO26-OBB and the local feature map output by Strip R-CNN to eliminate the scale difference between the two types of feature maps. Step S320: The normalization process and feature dimension alignment process are used to perform correlation modeling of the global feature map and the local feature map. In the spatial dimension, the second candidate feature is determined based on the material distribution characteristics and dopant morphology feature map, and invalid features are suppressed. Subsequently, the Mamba temporal fusion unit is used to perform pixel-by-pixel and region-by-region correlation modeling on the two types of features, to explore the correlation between global material distribution features and local dopant morphology features, and to suppress invalid feature information caused by background noise and illumination interference. Then, through an adaptive attention weighting mechanism, the global semantic features and local fine morphology features of effective dopant are weighted and strengthened, while the redundant features of normal materials are weakened. Step S330: Based on the attention mechanism, the feature weighted concatenation outputs a multimodal fusion feature map based on the first and second candidate features. Finally, through feature concatenation, convolution dimensionality reduction, and activation optimization, a multimodal fusion feature with both global semantic integrity and local morphological refinement is generated. This feature simultaneously contains multi-dimensional information such as color, texture, global distribution, and local shape.

[0041] Specifically, firstly, the multi-scale global feature map output by YOLO26-OBB and the refined local feature map output by Strip R-CNN are scaled, channel normalized, and dimension aligned to eliminate scale and channel differences between the two types of features. Then, a Mamba temporal fusion unit is connected, leveraging the long-distance modeling advantage of the state-space model to correlate global material distribution features and local dopant morphology features pixel-by-pixel and region-by-region, uncovering the coupling relationship between global semantics and local details, and adaptively suppressing invalid noise features caused by uneven illumination, material stacking, and background noise. Next, a channel-space dual attention weighting mechanism is introduced to weight and strengthen the effective global semantic features and local refined morphological features corresponding to dopants, while weakening redundant features of normal materials. Finally, through feature channel concatenation, convolutional dimensionality reduction, and activation function optimization, deep fusion of the two types of features is achieved, generating multimodal features with global semantic integrity, rotational pose accuracy, and local morphological refinement, significantly improving the utilization rate of effective features and providing high-quality feature input for subsequent multi-model shape detection.

[0042] S32 Based on the shape detection model, the multimodal features are processed to obtain the category confidence of the classification category; S33 Based on the comparison result between the category confidence score and the preset category confidence score threshold, an automatic rejection execution signal is generated, wherein the automatic rejection execution signal contains the coordinate information of the dopant.

[0043] In a recommended implementation, the process of training the automatic elimination model includes a first-stage training and a second-stage training. The first-stage training is pre-trained on the COCO-OBB rotating object detection dataset, and the second-stage training is fine-tuned on the detection dataset, freezing the backbone network and training only the shape detection model and the Mamba-YOLO fusion model.

[0044] Specifically, the automatic elimination model employs a two-stage training strategy for training and optimization. The first stage involves pre-training on the COCO-OBB rotating object detection dataset to teach the model basic rotating object detection capabilities. The second stage involves fine-tuning on the dopant detection dataset, freezing the backbone network, and training only the detection head and feature fusion layer to reduce training overhead and improve training accuracy.

[0045] During training, a core composite loss function is used to balance multiple optimization objectives such as category classification, shape regression, and color constraints, ensuring that the model accurately identifies impurities. The core formula is as follows:

[0046] Among them, L cls L represents the category classification loss, used to constrain the model's accuracy in classifying normal materials and dopants. confL represents the confidence loss, used to optimize the target detection confidence output and reduce invalid low-confidence predictions. shape L represents the shape regression loss, constraining the morphological parameter fitting accuracy of the three types of shape detection models. color L represents the color feature constraint loss, which helps distinguish materials with similar colors from dopants. fourie λ1, λ2, λ3, λ4, and λ5 represent the Fourier contour reconstruction loss, optimizing the contour restoration accuracy of incomplete and occluded targets. λ1, λ2, λ3, λ4, and λ5 represent their respective weighting coefficients. In a recommended implementation, the preferred values ​​for the weighting coefficients are λ1=1.0, λ2=1.0, λ3=2.5, λ4=0.5, and λ5=1.5.

[0047] This formula is the core composite loss function for model training. It is used to balance multi-dimensional optimization objectives such as category classification, shape regression, and color constraints to ensure that the model accurately identifies green stems. The preferred weight coefficients are λ1=1.0, λ2=1.0, λ3=2.5, λ4=0.5, and λ5=1.5.

[0048] The model is optimized through TensorRT-LLM quantization, cuDLA acceleration, and model pruning techniques to achieve model lightweighting, reduce inference latency, and ensure that the model can be deployed on edge devices to meet industrial real-time requirements.

[0049] In one specific embodiment provided in this application, the method is applied to remove green stems from tea products. During the training phase, a two-stage training strategy is employed: a pre-training phase involving 50 epochs on the COCO-OBB dataset, and a fine-tuning phase involving 200 epochs on the green stem dataset. A core composite loss function is used, with optimized weight coefficients. TensorRT-LLM quantization and cuDLA acceleration are employed to achieve lightweight model design and low-latency inference.

[0050] The preprocessed tea leaf image is input into the model, with an inference latency of 4.2ms and a detection accuracy of 99.1%. The model outputs the position coordinates and geometric parameters of the green stem, and accurately removes the green stem when it reaches the removal position, thus separating the tea leaves from the green stem.

[0051] like Figure 4 As shown, this application provides an automatic dopant removal device based on multimodal shape perception, including: a material image acquisition module, an image preprocessing module, a multimodal feature extraction module, a constructed multimodal feature fusion network, a shape detection module, and a confidence judgment module.

[0052] The material image acquisition module is used to acquire material images. The image preprocessing module is used to preprocess the material image to obtain a preprocessed image; A multimodal feature extraction module is used to extract multimodal features from the preprocessed image; wherein, "extracting multimodal features from the preprocessed image" includes: extracting multimodal features from the preprocessed image based on a constructed multimodal feature fusion network, wherein the multimodal feature fusion network adopts a fusion architecture of YOLO26-OBB, StripR-CNN, and Mamba-YOLO; wherein, YOLO26-OBB is used to extract deep semantic features of the image to obtain a global feature map; StripR-CNN is used to extract predetermined local features to obtain a local feature map; and Mamba-YOLO is used to fuse the global feature map and the local feature map to obtain the multimodal features.

[0053] The shape detection module is used to process the multimodal features based on the shape detection model to obtain the category confidence of the classification category; The confidence judgment module is used to generate an automatic rejection execution signal based on the comparison result between the category confidence and the preset category confidence threshold. The automatic rejection execution signal contains the coordinate information of the dopant.

[0054] The working process and working principle of each unit and module in the above-mentioned automatic dopant removal device based on multimodal shape perception are similar to those of the automatic dopant removal method based on multimodal shape perception described above, and will not be repeated here.

[0055] In another embodiment of this application, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, can implement the multimodal shape-aware automatic dopant removal method described above.

[0056] Obviously, the above embodiments of this application are merely examples for clearly illustrating this application, and are not intended to limit the implementation of this application. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for automatic dopant removal based on multimodal shape perception, characterized in that, Includes the following steps: S10 Acquire material image; S20 preprocesses the material image to obtain a preprocessed image; S30 is a trained automatic elimination model. It takes a pre-processed image as input, generates an automatic elimination execution signal, and the automatic elimination model executes the automatic elimination steps. The automatic elimination model includes the following process: S31 Extract multimodal features from the preprocessed image; S32 Based on the shape detection model, the multimodal features are processed to obtain the category confidence of the classification category; S33 Based on the comparison result between the category confidence score and the preset category confidence score threshold, an automatic rejection execution signal is generated, wherein the automatic rejection execution signal contains the coordinate information of the dopant; The step of "extracting multimodal features from the preprocessed image" includes: Based on the constructed multimodal feature fusion network, multimodal features are extracted from the preprocessed image. The multimodal feature fusion network adopts a fusion architecture of YOLO26-OBB, Strip R-CNN and Mamba-YOLO. The YOLO26-OBB is used to extract deep semantic features of images and obtain a global feature map. The Strip R-CNN is used to extract predetermined local features and obtain local feature maps; The Mamba-YOLO is used to fuse the global feature map and the local feature map to obtain the multimodal features.

2. The automatic dopant removal method based on multimodal shape perception as described in claim 1, characterized in that, "The YOLO26-OBB used to extract deep semantic features from images and obtain a global feature map" includes: The preprocessed image is processed using a low-dimensional feature fusion mechanism to obtain a dimensionality-reduced global feature map.

3. The automatic dopant removal method based on multimodal shape perception as described in claim 2, characterized in that, "The Strip R-CNN is used to extract predetermined local features and obtain local feature maps," including: The dimensionality-reduced global feature map output by YOLO26-OBB is obtained, and candidate morphological feature maps are extracted using a multi-scale sampling network; the morphological feature maps are used to describe the morphological characteristics of the dopant. If the morphological features meet the preset dopant morphology conditions, the morphological features of the dopant are identified as candidate features, and the candidate positions are determined through a region proposal network to suppress redundant features and output a local feature map based on the first candidate features.

4. The automatic dopant removal method based on multimodal shape perception as described in claim 3, characterized in that, "The Mamba-YOLO used to fuse the global feature map and the local feature map to obtain the multimodal features" includes: The global feature map output by YOLO26-OBB and the local feature map output by Strip R-CNN are processed by feature normalization and feature dimension alignment to obtain normalized global feature map and local feature map. Normalization and feature dimension alignment are performed to associate and model the global and local feature maps. In the spatial dimension, the second candidate feature is determined based on the material distribution characteristics and dopant morphology feature map, and invalid features are suppressed. The attention-based feature weighted concatenation outputs a multimodal fusion feature map based on the first and second candidate features.

5. The automatic dopant removal method based on multimodal shape perception as described in claim 1, characterized in that, The shape detection model is selected from at least one of the first shape detection model, the second shape detection model, and the third shape detection model; The first shape detection model includes an elliptic parametric regression model; the second shape detection model includes a key point contour fitting model; and the third shape detection model includes a Fourier series distance field shape perception model.

6. The automatic dopant removal method based on multimodal shape perception as described in claim 5, characterized in that, The shape detection model runs the first shape detection model, the second shape detection model, and the third shape detection model simultaneously.

7. The automatic dopant removal method based on multimodal shape perception as described in claim 1, 5, or 6, characterized in that, The process of training the automatic elimination model includes a first-stage training and a second-stage training. The first-stage training is pre-trained on the COCO-OBB rotating object detection dataset, and the second-stage training is fine-tuned on the detection dataset, freezing the backbone network and training only the shape detection model and the Mamba-YOLO fusion model.

8. The automatic dopant removal method based on multimodal shape perception as described in claim 1, characterized in that, In step S20, "preprocessing the material image to obtain a preprocessed image" includes: The material image is subjected to at least one of the following: distortion correction processing, white balance processing, and data enhancement processing.

9. An automatic dopant removal device based on multimodal shape sensing, characterized in that, include: The material image acquisition module is used to acquire material images; The image preprocessing module is used to preprocess the material image to obtain a preprocessed image; A multimodal feature extraction module is used to extract multimodal features from the preprocessed image; The shape detection module is used to process the multimodal features based on the shape detection model to obtain the category confidence of the classification category; The confidence level judgment module is used to generate an automatic rejection execution signal based on the comparison result between the category confidence level and the preset category confidence level threshold. The automatic rejection execution signal contains the coordinate information of the dopant. The step of "extracting multimodal features from the preprocessed image" includes: Based on the constructed multimodal feature fusion network, multimodal features are extracted from the preprocessed image. The multimodal feature fusion network adopts a fusion architecture of YOLO26-OBB, Strip R-CNN and Mamba-YOLO. The YOLO26-OBB is used to extract deep semantic features of images and obtain a global feature map. The Strip R-CNN is used to extract predetermined local features and obtain local feature maps; The Mamba-YOLO is used to fuse the global feature map and the local feature map to obtain the multimodal features.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement the automatic dopant removal method based on multimodal shape awareness as described in any one of claims 1-8.

Images