Multi-modal sensor and deep learning algorithm fusion-based bougainvillea speetabilis multi-color petal classification method, system and terminal

Abstract

The invention belongs to the technical field of agricultural information technology and computer vision crossing, and particularly relates to a bougainvillea speetabilis multi-color petal classification method and system based on fusion of a multi-modal sensor and a deep learning algorithm, and a terminal. Imaging parameters and illumination conditions are calibrated in real time in combination with a TOF distance sensor and an ambient light sensor; channel pruning and INT8 quantization are combined, so that the model obtains efficient running speed and memory occupation on the mobile terminal; the optimized MobileNetV3 model is utilized to extract edge features, and a Sobel operator and an HOG feature descriptor are matched to analyze the color gradient direction; an Adam optimization algorithm is adopted to train the model, and a classification threshold value is dynamically adjusted based on GPS geographic position data; outputting a visual classification result containing the thermodynamic diagram and the gradual change proportion; according to the invention, high-precision and light-weight real-time classification is realized.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of agricultural information technology and computer vision, specifically involving a method, system and terminal for classifying bougainvillea multicolored petals based on the fusion of multimodal sensors and deep learning algorithms. Background Technology

[0002] Bougainvillea, a common ornamental plant, plays a crucial role in horticultural variety identification through the identification of its multicolored petals. Traditional methods for classifying bougainvillea multicolored petals primarily utilize the HSV color space threshold segmentation method, which distinguishes gradual color transitions in petals by setting fixed saturation thresholds. However, this method performs poorly under varying outdoor lighting conditions. Specifically, fluctuations in lighting conditions easily lead to threshold drift, resulting in classification accuracy generally below 70%. This is especially true when there are significant differences in light intensity at different geographical latitudes, where the model's performance becomes particularly unstable, and classification accuracy decreases significantly.

[0003] In recent years, with the development of deep learning technology, researchers have begun to try using relevant models for plant classification, including various plant parts such as leaves. For example, some studies have proposed using the ResNet50 model to identify plant species, which has proven effective to some extent. However, these methods also have limitations. Due to the high complexity and large number of parameters, they are inadequate for real-time inference on mobile devices. Specifically, for the classification of bougainvillea's multi-color components, while existing deep learning models such as ResNet50 can improve classification accuracy to some extent, their large number of parameters and complex computational requirements (approximately 23.5MB) make real-time inference on mobile devices very difficult, especially on devices with limited computing resources such as Android devices, where the running speed is usually far below the ideal 15 frames per second.

[0004] Traditional saturation threshold segmentation methods rely too heavily on fixed thresholds and are poorly adaptable to changes in lighting conditions. Meanwhile, while deep learning models can significantly improve classification accuracy, their practical application on mobile devices is limited by model complexity and computational resources. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, and terminal for classifying bougainvillea multi-colored petals based on the fusion of multimodal sensors and deep learning algorithms, so as to overcome the shortcomings of traditional bougainvillea multi-colored classification methods in terms of light sensitivity, computational complexity, and cross-regional adaptability, and to achieve high-precision, lightweight real-time classification.

[0006] The present invention achieves the above objectives through the following technical solutions: Firstly, this invention proposes a method for classifying the multi-colored petals of bougainvillea based on the fusion of multimodal sensors and deep learning algorithms, the method comprising: S1. Acquire multi-band image data of the polychromatic region of bougainvillea petals, and after performing illumination scene calibration and imaging parameter calibration based on real-time monitoring shooting distance, form a standardized feature image; wherein the polychromatic region is a visual feature region that presents two or more color distributions. S2. Use the MobileNetV3 pre-trained model to extract edge features of the normalized feature image, and use the Sobel operator to calculate the image gradient magnitude of the normalized feature image. Then, use the HOG feature descriptor to analyze the direction features of color gradation and form a composite feature vector. S3. The MobileNetV3 pre-trained model optimized by the Adam optimization algorithm is used to process the composite feature vector and output the information on the types of bougainvillea polychrome and the proportion of each type of color gradient region.

[0007] Furthermore, step S1 specifically includes: S1.1 Obtain preliminary multispectral images of the polychromatic areas of bougainvillea petals under visible light, near-infrared, red-edge, and blue-violet filters; S1.2. Dynamically adjust the light intensity and angle information under the current lighting environment based on the real-time detected ambient light intensity, adjust the shooting focal length based on the real-time monitored shooting distance, and dynamically adjust the image sampling frequency based on the distance change rate to form the standardized feature image.

[0008] Furthermore, step S2 specifically includes: S2.1 Input the standardized feature image into the MobileNetV3 network that has undergone channel pruning, compress the output layer dimension of the network from 1280 dimensions to 512 dimensions, extract features through multi-scale convolution kernels, and use max pooling layers to reduce the dimensionality of the extracted features to obtain edge features; S2.2. The Sobel operator is used to calculate the gradient components in the horizontal and vertical directions of the image respectively. The magnitude and orientation angle of the gradient components are calculated. The orientation feature analysis is performed based on the HOG feature descriptor to generate a feature vector describing the direction of color gradient. S2.3 Normalize the edge features and gradient feature vectors, and generate the composite feature vector by feature splicing or weighted fusion.

[0009] Furthermore, in step S2, before extracting edge features of the standardized feature image using the MobileNetV3 pre-trained model, the method further includes: A transfer learning strategy was used to initialize the parameters of the MobileNetV3 pre-trained model, freeze the parameters of the bottom convolutional layers, and fine-tune the top three convolutional layers. The MobileNetV3 pre-trained model was trained using the PlantCLEF plant dataset and a self-collected bougainvillea image dataset. The self-collected bougainvillea image dataset contains multispectral images of the polychromatic regions of bougainvillea petals collected under different lighting conditions.

[0010] Furthermore, the method also includes dynamically adjusting the saturation threshold of the polychrome area of ​​the bougainvillea by combining the latitude and longitude data of the bougainvillea plants obtained in advance, specifically including: Based on the latitude information in the latitude and longitude data, the saturation threshold is dynamically adjusted according to the preset latitude-threshold mapping relationship; By combining real-time ambient light intensity data, when a specific lighting condition is detected, a corresponding optical compensation mechanism is triggered; the specific lighting condition includes light intensity meeting a preset range.

[0011] Furthermore, the method also includes: converting the information on the types of bougainvillea polychrome and the proportion of each color gradient region into visual information for display. The visual information includes a gradient heat map generated using color space mapping, and a classification result map labeled with polychrome type identifiers and gradient region proportions. The gradient heatmap includes the color gradient trend rendered by the color space module, and the classification result map includes the step transition trend after saturation grading.

[0012] Secondly, this invention proposes a bougainvillea multi-colored petal classification system based on the fusion of multimodal sensors and deep learning algorithms, applied to the method described above. The system includes: The image acquisition module is used to acquire multi-band image data of the polychromatic region of bougainvillea petals, and after performing illumination scene calibration and imaging parameter calibration based on real-time monitoring shooting distance, it forms a standardized feature image; wherein the polychromatic region is a visual feature region that presents two or more color distributions. The feature extraction module is used to extract edge features of the normalized feature image using the MobileNetV3 pre-trained model, calculate the image gradient magnitude of the normalized feature image using the Sobel operator, and analyze the direction features of color gradation using the HOG feature descriptor to form a composite feature vector. The classification processing module is used to process the composite feature vector using the MobileNetV3 pre-trained model optimized by the Adam optimization algorithm, perform channel pruning and INT8 quantization, dynamically adjust the learning rate through the cosine annealing strategy, and output information on the types of bougainvillea polychrome and the proportion of each color gradient region. The display module is used to convert the information on the types of bougainvillea polychrome and the proportion of each color gradient area into visual information for display. The visual information includes a gradient heat map of color space mapping, as well as a classification result map labeled with polychrome type identifiers and gradient area proportions.

[0013] Furthermore, the system also includes a classification optimization module, which is used to dynamically adjust the saturation threshold of the bougainvillea polychrome area based on the pre-acquired latitude and longitude data of the bougainvillea plants. Specifically, this includes: dynamically adjusting the saturation threshold according to a preset latitude-threshold mapping relationship based on the latitude information in the latitude and longitude data; and triggering a corresponding optical compensation mechanism when a specific lighting condition is detected, based on real-time ambient light intensity data. The specific lighting condition includes light intensity meeting a preset range.

[0014] Furthermore, the feature extraction module includes: The edge feature extraction unit is used to input the standardized feature image into the channel-pruned MobileNetV3 network, compress the network output layer dimension from 1280 dimensions to 512 dimensions, extract features through multi-scale convolution kernels, and use max pooling layers to reduce the dimensionality of the extracted features to obtain edge features. The gradient analysis unit is used to calculate the gradient components in the horizontal and vertical directions of the image using the Sobel operator, calculate the magnitude and orientation angle of the gradient components, perform orientation feature analysis based on the HOG feature descriptor, and generate a feature vector describing the direction of color gradient. The feature fusion unit is used to normalize the edge features and gradient feature vectors, and generate the composite feature vector by feature concatenation or weighted fusion.

[0015] Thirdly, the present invention proposes a terminal, comprising: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, enable the terminal to implement the bougainvillea multi-colored petal classification method described above.

[0016] The beneficial effects of this invention are as follows: This invention achieves a systematic improvement in bougainvillea polycolor classification technology through the synergy of multimodal sensors and lightweight deep learning. At the technical implementation level, multispectral imaging combined with dynamic light source compensation effectively overcomes the impact of outdoor light fluctuations on color recognition, enabling the classification system to maintain stable performance under different lighting conditions. The MobileNetV3 model, optimized through channel pruning and quantization, significantly reduces computational resource requirements, allowing complex deep learning models to run efficiently on mobile terminals. A geographic location-adaptive threshold adjustment mechanism is introduced, dynamically optimizing classification parameters using GPS data, enhancing the system's adaptability to changes in different geographical environments. At the application level, this scheme achieves more refined feature characterization of the polycolor regions of petals through multi-level fusion of edge features and gradient direction features. The output heatmap and classification results intuitively demonstrate the spatial distribution characteristics of color gradients. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a method for classifying bougainvillea multi-colored petals based on the fusion of multimodal sensors and deep learning algorithms, provided in one embodiment of this application; Figure 2 This is another flowchart illustrating a method for classifying bougainvillea multi-colored petals based on the fusion of multimodal sensors and deep learning algorithms, provided as an embodiment of this application. Figure 3 This is a system block diagram of a bougainvillea multi-colored petal classification system based on the fusion of multimodal sensors and deep learning algorithms, provided as an embodiment of this application. Detailed Implementation

[0018] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0019] It is worth noting that the bougainvillea petal polychrome areas that need to be classified in this invention exhibit the following color characteristics: gradient transition type: such as a continuous color gradient from the base (red) to the edge (yellow); boundary differentiation type: such as a clear dividing line between purple and white in a local area of ​​the petal; texture mixing type: such as spots or stripes of different colors distributed on a base color. The color distribution characteristics of this area are formed by the non-uniform distribution of pigments such as anthocyanins and carotenoids, as well as the optical effects of epidermal cell structure, and are key phenotypic indicators for variety identification.

[0020] Example 1 Please see Figure 1 and Figure 2This application proposes a method for classifying the multi-colored petals of bougainvillea based on the fusion of multimodal sensors and deep learning algorithms. The method includes the following steps: S1. Acquire multi-band image data of the polychromatic region of bougainvillea petals, and after performing illumination scene calibration and imaging parameter calibration based on real-time monitoring shooting distance, form a standardized feature image; wherein the polychromatic region is a visual feature region that presents two or more color distributions, including but not limited to color gradient transition area, boundary differentiation area and texture mixing area.

[0021] During image acquisition, to cope with complex and ever-changing outdoor lighting environments, the system monitors ambient lighting conditions in real time. For example, when the detected light intensity exceeds 10,000 lux, a backlight compensation algorithm is automatically triggered, optimizing imaging conditions by adjusting the light intensity and angle (adjustable range of 30°-60°) of the ring light source array. Simultaneously, based on TOF ranging technology, real-time shooting distance data is acquired. When the distance exceeds the optimal range of 30-50cm, the imaging focal length and sampling frequency are automatically adjusted to ensure the acquisition of stable, standardized feature images.

[0022] More specifically, step S1 includes: S1.1 The multispectral camera sequentially switches between filters of different wavelengths to acquire preliminary multispectral images of the polychromatic areas of bougainvillea petals under visible light, near-infrared, red-edge, and blue-violet light filters. S1.2. Dynamically adjust the light intensity and angle information under the current lighting environment based on the real-time detected ambient light intensity, adjust the shooting focal length based on the real-time monitored shooting distance (e.g., use a TOF depth sensor to monitor the shooting distance in real time and feed it back to the system), and dynamically adjust the image sampling frequency based on the distance change rate to form a standardized feature image.

[0023] In the image preprocessing stage, white balance correction and color calibration are performed on the acquired multi-band image data. Specifically, this includes: dynamically adjusting the image's white balance parameters according to ambient lighting conditions; performing geometric correction on the image based on shooting distance data; and adaptively adjusting the image sampling frequency according to the rate of change of distance. Through these processes, a feature image with a unified standard is ultimately formed, providing high-quality input data for subsequent feature extraction and classification.

[0024] S2. Use the MobileNetV3 pre-trained model (lightweight convolutional neural network architecture) to extract edge features of the normalized feature image, and use the Sobel (Sobel Operator) operator (edge ​​detection operator) to calculate the image gradient magnitude of the normalized feature image. Then, use the HOG (Histogram of Oriented Gradients) feature descriptor (an algorithm that describes image features by statistically analyzing the gradient direction distribution of local regions) to analyze the directional features of color gradation and form a composite feature vector.

[0025] More specifically, step S2 includes: S2.1 Input the standardized feature image into the MobileNetV3 network that has undergone channel pruning, compress the output layer dimension of the network from 1280 dimensions to 512 dimensions, extract features through multi-scale convolution kernels, and use max pooling layers to reduce the dimensionality of the extracted features to obtain edge features; S2.2. The Sobel operator is used to calculate the gradient components in the horizontal and vertical directions of the image respectively. The magnitude and orientation angle of the gradient components are calculated. The orientation feature analysis is performed based on the HOG feature descriptor. The image is divided into cells of a set number of pixels. The gradient orientation histogram of each orientation interval is calculated. The histogram is normalized with cells as blocks. The gradient intensity distribution of each orientation interval is statistically analyzed to generate a feature vector describing the direction of color gradient. S2.3 Normalize the edge features and gradient feature vectors, and generate composite feature vectors by feature splicing or weighted fusion.

[0026] Understandably, this invention employs a multi-feature fusion strategy in the feature extraction stage to comprehensively characterize the visual features of bougainvillea's multicolored petals. First, the standardized feature image is input into a specially optimized MobileNetV3 deep neural network for feature extraction. This network undergoes channel pruning, compressing the original 1280 feature channels of the output layer to 512 by evaluating the importance of each channel, significantly reducing model complexity while maintaining feature expressiveness. The network uses multi-scale convolutional kernels (including 3×3, 5×5, and 7×7 sizes) to capture edge features of different granularities, and then performs dimensionality reduction on the feature map using a 2×2 max-pooling layer, reducing computational cost while preserving important features.

[0027] Preferably, in step S2, before extracting edge features of the standardized feature image using the MobileNetV3 pre-trained model, the method further includes: initializing the parameters of the MobileNetV3 pre-trained model using a transfer learning strategy, freezing the parameters of the bottom convolutional layers, and fine-tuning the training of the top three convolutional layers; training the MobileNetV3 pre-trained model using the PlantCLEF plant dataset and a self-collected bougainvillea image dataset; wherein, the self-collected bougainvillea image dataset contains multispectral images of the polychromatic regions of bougainvillea petals collected under different lighting conditions.

[0028] In step S2 above, the transfer learning method is used to optimize MobileNetV3 for model training. Specifically, the parameters of the bottom convolutional layers (conv1 to conv15) are first frozen to maintain their general feature extraction capabilities learned on the ImageNet dataset; then, the top three convolutional layers (conv16 to conv18) are fine-tuned to adapt to the specific classification task of bougainvillea polychrome. The training data combines the PlantCLEF general plant dataset and a specially collected dataset of bougainvillea polychrome images. The bougainvillea dataset consists of multispectral images collected under different lighting conditions to ensure the model has good generalization ability.

[0029] Simultaneously, image processing methods are employed to supplement the analysis of color gradient features. Specifically, a 3×3 Sobel operator is used to calculate the gradient components of the image in the horizontal and vertical directions, thereby obtaining the gradient magnitude and orientation angle of each pixel. Based on this gradient information, the HOG (Histogram of Oriented Gradients) feature descriptor is applied for more in-depth directional feature analysis: for example, the image is divided into 8×8 pixel cells, and the gradient histograms in 9 directions are calculated in each cell; then, histogram normalization is performed in 2×2 cell blocks, and finally, the gradient intensity distribution in each direction is statistically analyzed to form a feature vector describing the direction of color gradient.

[0030] In the feature fusion stage, the edge features extracted by the deep network and the obtained gradient direction features are normalized to eliminate the dimensional differences between different features. Then, the two types of features are combined into a composite feature vector by feature concatenation. If necessary, an attention mechanism can be introduced to perform weighted fusion of features to highlight the role of important features.

[0031] S3. The MobileNetV3 pre-trained model optimized by the Adam optimization algorithm is used to process the composite feature vector and output the information on the types of bougainvillea colors and the proportion of each color gradient region.

[0032] The Adam optimization algorithm effectively improves the model's convergence speed and classification accuracy by adaptively adjusting the learning rate (e.g., setting the initial value to 0.001) and combining dynamic calculations of first-order moment estimation and second-order moment estimation.

[0033] Preferably, the method further includes dynamically adjusting the saturation threshold of the bougainvillea polychrome area based on the latitude and longitude data of the bougainvillea plants obtained in advance, specifically including: dynamically adjusting the saturation threshold according to the latitude information in the latitude and longitude data and a preset latitude-threshold mapping relationship; and triggering a corresponding optical compensation mechanism when a specific lighting condition is detected, based on real-time ambient light intensity data; the specific lighting condition includes light intensity meeting a preset range.

[0034] Understandably, to enhance adaptability to different geographical environments, this application introduces GPS geographic location data to assist classification decisions. In specific implementation, the latitude and longitude information of the plant's location is acquired in real time, establishing a dynamic mapping relationship between latitude and saturation thresholds: for example, when the absolute latitude is less than 20 degrees, the basic saturation threshold is automatically lowered by 5%-8% to cope with strong light conditions; simultaneously, combined with real-time monitored ambient light intensity data, a backlight compensation mechanism is triggered when the light intensity exceeds 10,000 lux, compensating for light interference by adjusting the classifier's decision boundary. This adaptive strategy significantly improves the system's classification stability in low-latitude regions and high-light environments.

[0035] It is important to emphasize that this invention significantly improves the accuracy and environmental adaptability of bougainvillea polychromatic color classification by dynamically adjusting the saturation classification threshold in the HSV color space. This technology addresses complex outdoor lighting conditions, particularly the differences in light intensity across different latitudes, by establishing a dynamic mapping mechanism between latitude and saturation thresholds: when the GPS module detects a latitude below 20°, the system automatically lowers the base saturation threshold by 5%-8%, effectively compensating for color oversaturation caused by strong light. This adaptive adjustment enables the classifier to accurately distinguish between true color gradations and lighting artifacts. For example, in low-latitude regions, the system can retain weak saturation gradation features that might otherwise be filtered out (such as subtle transitions from light pink to pale yellow) by lowering the threshold. Simultaneously, under high light conditions (>10000 lux), the backlight compensation algorithm further optimizes the feature extraction process.

[0036] As a preferred embodiment, the method further includes: converting the information on the types of bougainvillea polychrome and the proportion of each type of color gradient region into visual information for display. The visual information includes a gradient heatmap generated using color space mapping, and a classification result map labeled with polychrome type identifiers and gradient region proportions. The gradient heatmap includes the color gradient trend rendered by the color space module, and the classification result map includes a step-like transition trend after saturation grading.

[0037] As an example, two complementary presentation methods are used for result visualization: one is a gradient heatmap generated based on HSV color space mapping, which visually displays the color transition trend by dividing the saturation gradient into 8 discrete levels and using color gradients (such as from deep red to light yellow); the other is a classification result chart labeled with compound color type identifiers (such as "red-yellow gradient") and precise proportion data (such as "red area accounts for 65%)", in which step-like color blocks are used to distinguish different saturation ranges. The visualization module outputs in real time through an OLED display (such as a resolution of 320×240 and a refresh rate of 60Hz).

[0038] According to the above embodiments of this application, the present invention, based on the technical principle of multimodal sensing and deep learning fusion, achieves high-precision bougainvillea polychromatic classification through systematic collaborative optimization. Its core technical process is as follows: First, multi-dimensional image data of the petal polychromatic region is acquired using a multispectral imaging system (visible / near-infrared / red-edge / blue-violet bands). Imaging parameters and lighting conditions are calibrated in real-time using TOF ranging and an ambient light sensor to form a standardized feature image. In the feature extraction stage, a channel-pruned optimized MobileNetV3 network (output layer 1280→512 dimensions) is used to extract multi-scale edge features. Simultaneously, the Sobel operator and HOG descriptor are used to analyze color gradient direction features, resulting in a composite feature vector after feature fusion. In the classification decision stage, a geographic adaptive mechanism is introduced, dynamically adjusting the HSV saturation threshold based on GPS latitude information (e.g., lowering the threshold by 5-8% when latitude <20°). The Adam optimization algorithm (initial learning rate 0.001, cosine annealing strategy) is combined to optimize model parameters, ultimately outputting a classification result including a gradient heatmap (8 levels of HSV color space) and quantization percentage. This solution achieves accurate capture and analysis of color gradient features in complex environments through the synergistic use of three technologies: multispectral imaging compensation, lightweight model optimization, and geographic adaptive classification. It solves the core problems faced by traditional methods when deployed on mobile devices, such as light sensitivity, computational complexity, and poor regional adaptability.

[0039] Example 2 Please see Figure 3 This application proposes a bougainvillea multi-colored petal classification system based on the fusion of multimodal sensors and deep learning algorithms, which is applied to perform the bougainvillea multi-colored petal classification method as proposed in Example 1. The system includes an image acquisition module, a feature extraction module, a classification processing module, a classification optimization module, and a display module.

[0040] The image acquisition module is used to acquire multi-band image data of the polychromatic area of ​​bougainvillea petals, and after performing illumination scene calibration and imaging parameter calibration based on real-time monitoring shooting distance, it forms a standardized feature image; the polychromatic area is a visual feature area that presents two or more color distributions.

[0041] More specifically, the image acquisition module includes a multispectral imaging device (such as a multispectral camera) and a distance sensing unit (such as a TOF depth sensor), configured for autofocus at a shooting distance of 30-50cm. The multispectral camera acquires multispectral images of the polychromatic areas of the bougainvillea. The multispectral camera internally houses a filter switching mechanism, which includes a rotating disk and a drive motor. Four filters of different wavelengths are equidistantly mounted on the rotating disk: visible light, near-infrared, red-edge, and blue-violet. The optical transmittance of each filter remains above 90% within its corresponding wavelength range. The drive motor is a stepper motor with a step angle of 1.8°, ensuring the rotating disk rotates 90° each time a switch is made. The TOF depth sensor monitors the shooting distance in real time and feeds it back to the system.

[0042] The image acquisition module also includes an ambient light compensation module, comprising a ring light source array and an ambient light detection component. The ring light source array consists of six independently controllable LED light sources, evenly distributed around the periphery of the multispectral camera. Each LED light source has an independent dimming circuit and an independent scattering lens. The scattering lens adopts a Fresnel lens structure, with micron-level concentric circular patterns on its surface, which can evenly disperse light to the shooting area. The emission angle of each LED light source is adjustable from 30° to 60°. The ambient light detection component includes an LX1972 photosensitive chip and an angle adjustment bracket. The LX1972 photosensitive chip is fixed above the multispectral camera via the angle adjustment bracket, with the detection surface facing the center of the shooting area. The emission intensity of each LED light source is dynamically adjusted based on the ambient light intensity data acquired by the LX1972 photosensitive chip.

[0043] The feature extraction module is used to extract edge features of the normalized feature image using the MobileNetV3 pre-trained model, calculate the image gradient magnitude of the normalized feature image using the Sobel operator, and analyze the direction features of color gradation using the HOG feature descriptor to form a composite feature vector.

[0044] Preferably, the feature extraction module includes: The edge feature extraction unit is used to input the standardized feature image into the channel-pruned MobileNetV3 network, compress the network output layer dimension from 1280 dimensions to 512 dimensions, extract features through multi-scale convolution kernels, and use max pooling layers to reduce the dimensionality of the extracted features to obtain edge features.

[0045] As an example, the edge feature extraction unit extracts edge features from multispectral images using a MobileNetV3 pre-trained model. The model's output layer is compressed from 1280 dimensions to 512 dimensions through channel pruning, reducing the number of parameters by 57%. The model employs a transfer learning strategy, freezing the bottom convolutional layers (conv1-15) and only fine-tuning the top three layers (conv16-18). Cross-domain training is conducted using the PlantCLEF dataset and 500 self-collected bougainvillea images. The edge feature extraction unit also includes a convolution operation module and a feature mapping module. The convolution operation module has three convolutional kernels of different scales, with sizes of 3×3, 5×5, and 7×7, and the weight parameters of each convolutional kernel are optimized and determined through a neural architecture search method. The feature mapping module uses max pooling to reduce feature dimensions while retaining important information, with a pooling window size of 2×2 and a stride of 2.

[0046] The gradient analysis unit is used to calculate the gradient components in the horizontal and vertical directions of the image using the Sobel operator, calculate the magnitude and orientation angle of the gradient components, perform orientation feature analysis based on the HOG feature descriptor, and generate a feature vector describing the direction of color gradient.

[0047] As an example, the gradient detection unit is connected to the edge enhancement unit via a data transmission line, and also includes a gradient calculation module and a direction analysis module. The gradient calculation module uses the Sobel operator to calculate gradients, and the convolution kernel size in both the horizontal and vertical directions is 3×3. The direction analysis module uses the HOG feature descriptor to analyze the direction features of the color gradient, with a cell size of 8×8 pixels and a block size of 2×2 cells. The extracted feature vector is input to the classification processing module.

[0048] The feature fusion unit is used to normalize edge features and gradient feature vectors, and generate composite feature vectors through feature concatenation or weighted fusion.

[0049] The classification processing module is used to process the composite feature vector using the MobileNetV3 pre-trained model optimized by the Adam optimization algorithm. This includes channel pruning and INT8 quantization, dynamically adjusting the learning rate through a cosine annealing strategy, and outputting information on the types of bougainvillea polychromes and the proportion of each color gradient region. The INT8 quantization includes a dynamic truncation threshold calibration mechanism, with an error compensation rate set to 3%.

[0050] As an example, the parameters of the MobileNetV3 pre-trained model are optimized using the Adam optimization algorithm, with an initial learning rate of 0.001, an exponential decay rate of 0.9 for the first moment estimation, and an exponential decay rate of 0.999 for the second moment estimation. The model also employs a cosine annealing strategy with a period length of 10 epochs.

[0051] The classification optimization module is used to dynamically adjust the saturation threshold of the polychrome area of ​​bougainvillea by combining the latitude and longitude data of the bougainvillea plants obtained in advance. Specifically, it includes: dynamically adjusting the saturation threshold according to the latitude information in the latitude and longitude data and the preset latitude-threshold mapping relationship; and triggering the corresponding optical compensation mechanism when a specific lighting condition is detected by combining real-time ambient light intensity data. The specific lighting condition includes the light intensity meeting the preset range.

[0052] As an example, the classification optimization module acquires GPS latitude and longitude data in real time and adjusts the saturation threshold. For instance, the threshold is lowered by 2% for every 10° decrease in latitude. The rules for adjusting the saturation threshold include: when the latitude is less than 20°, the threshold is lowered by 5%; when the light intensity is greater than 10,000 lux, the backlight compensation algorithm is triggered.

[0053] The display module is used to convert the information on the types of bougainvillea polychrome and the proportion of each color gradient area into visual information for display. The visual information includes a gradient heat map of color space mapping, as well as a classification result map labeled with polychrome type identifiers and gradient area proportions.

[0054] Specific limitations regarding the bougainvillea multi-colored petal classification system based on the fusion of multimodal sensors and deep learning algorithms can be found in the limitations of the bougainvillea multi-colored petal classification method based on the fusion of multimodal sensors and deep learning algorithms described above, and will not be repeated here. It should be noted that each module in the above classification system corresponds to a step in implementing the above classification method. Multiple modules and their corresponding steps may have the same implementation examples and application scenarios, but are not limited to the content disclosed in Embodiment 1 above.

[0055] Example 3 This application also proposes a terminal, including: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, enable the terminal to implement the bougainvillea multi-colored petal classification method as described in Example 1.

[0056] It is understood that the terminal device proposed in this invention can be specifically implemented as an intelligent horticultural detector or a mobile plant analysis terminal. It integrates a multispectral imaging module (including visible / near-infrared / red-edge / blue-violet four-band filters), a TOF ranging unit, and an ambient light sensor, and executes a classification algorithm through a built-in processor. In use, the operator only needs to point the terminal at the bougainvillea petals and maintain a shooting distance of 30-50cm; the system automatically completes multispectral image acquisition, ambient light compensation, and distance calibration. The terminal displays the classification results in real time, including: 1) a gradient heatmap rendered in the HSV color space, visually presenting the color transition trend in 8 color levels; 2) a classification result diagram indicating the type of compound color (e.g., "red-yellow gradient") and the proportion of each region.

[0057] This terminal is particularly suitable for field variety identification. When GPS positioning indicates a low latitude region (<20°), it automatically activates a strong light compensation mode to ensure classification accuracy under complex outdoor lighting conditions. Data can be synchronized to the cloud via a Type-C interface or wireless transmission to establish a bougainvillea variety database, providing data support for horticultural research and commercial breeding.

[0058] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated.

[0059] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0060] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0061] In addition, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0062] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for classifying the multi-colored petals of bougainvillea based on the fusion of multimodal sensors and deep learning algorithms, characterized in that, The method includes: S1. Acquire multi-band image data of the polychromatic region of bougainvillea petals, and after performing illumination scene calibration and imaging parameter calibration based on real-time monitoring shooting distance, form a standardized feature image; wherein the polychromatic region is a visual feature region that presents two or more color distributions. S2. Use the MobileNetV3 pre-trained model to extract edge features of the normalized feature image, and use the Sobel operator to calculate the image gradient magnitude of the normalized feature image. Then, use the HOG feature descriptor to analyze the direction features of color gradation and form a composite feature vector. S3. The MobileNetV3 pre-trained model optimized by the Adam optimization algorithm is used to process the composite feature vector and output the information on the types of bougainvillea polychrome and the proportion of each type of color gradient region.

2. The bougainvillea multi-colored petal classification method based on the fusion of multimodal sensors and deep learning algorithms according to claim 1, characterized in that, Step S1 specifically includes: S1.1 Obtain preliminary multispectral images of the polychromatic areas of bougainvillea petals under visible light, near-infrared, red-edge, and blue-violet filters; S1.

2. Dynamically adjust the light intensity and angle information under the current lighting environment based on the real-time detected ambient light intensity, adjust the shooting focal length based on the real-time monitored shooting distance, and dynamically adjust the image sampling frequency based on the distance change rate to form the standardized feature image.

3. The bougainvillea multi-colored petal classification method based on the fusion of multimodal sensors and deep learning algorithms according to claim 1, characterized in that, Step S2 specifically includes: S2.1 Input the standardized feature image into the MobileNetV3 network that has undergone channel pruning, compress the output layer dimension of the network from 1280 dimensions to 512 dimensions, extract features through multi-scale convolution kernels, and use max pooling layers to reduce the dimensionality of the extracted features to obtain edge features; S2.

2. The Sobel operator is used to calculate the gradient components in the horizontal and vertical directions of the image respectively. The magnitude and orientation angle of the gradient components are calculated. The orientation feature analysis is performed based on the HOG feature descriptor to generate a feature vector describing the direction of color gradient. S2.3 Normalize the edge features and gradient feature vectors, and generate the composite feature vector by feature splicing or weighted fusion.

4. The bougainvillea multi-colored petal classification method based on the fusion of multimodal sensors and deep learning algorithms according to claim 1, characterized in that, In step S2, before extracting edge features of the standardized feature image using the MobileNetV3 pre-trained model, the method further includes: A transfer learning strategy was used to initialize the parameters of the MobileNetV3 pre-trained model, freeze the parameters of the bottom convolutional layers, and fine-tune the top three convolutional layers. The MobileNetV3 pre-trained model was trained using the PlantCLEF plant dataset and a self-collected bougainvillea image dataset. The self-collected bougainvillea image dataset contains multispectral images of the polychromatic regions of bougainvillea petals collected under different lighting conditions.

5. The bougainvillea multi-colored petal classification method based on the fusion of multimodal sensors and deep learning algorithms according to claim 1, characterized in that, The method also includes dynamically adjusting the saturation threshold of the polychrome area of ​​the bougainvillea by combining the latitude and longitude data of the bougainvillea plants obtained in advance, specifically including: Based on the latitude information in the latitude and longitude data, the saturation threshold is dynamically adjusted according to the preset latitude-threshold mapping relationship; By combining real-time ambient light intensity data, when a specific lighting condition is detected, a corresponding optical compensation mechanism is triggered; the specific lighting condition includes light intensity meeting a preset range.

6. The bougainvillea multi-colored petal classification method based on the fusion of multimodal sensors and deep learning algorithms according to claim 1, characterized in that, The method further includes: converting the information on the types of bougainvillea polychrome and the proportion of each color gradient region into visual information for display. The visual information includes a gradient heat map generated using color space mapping, and a classification result map labeled with polychrome type identifiers and gradient region proportions. The gradient heatmap includes the color gradient trend rendered by the color space module, and the classification result map includes the step transition trend after saturation grading.

7. A bougainvillea multi-colored petal classification system based on the fusion of multimodal sensors and deep learning algorithms, applied to the method described in any one of claims 1-6, characterized in that, The system includes: The image acquisition module is used to acquire multi-band image data of the polychromatic region of bougainvillea petals, and after performing illumination scene calibration and imaging parameter calibration based on real-time monitoring shooting distance, it forms a standardized feature image; wherein the polychromatic region is a visual feature region that presents two or more color distributions. The feature extraction module is used to extract edge features of the normalized feature image using the MobileNetV3 pre-trained model, calculate the image gradient magnitude of the normalized feature image using the Sobel operator, and analyze the direction features of color gradation using the HOG feature descriptor to form a composite feature vector. The classification processing module is used to process the composite feature vector using the MobileNetV3 pre-trained model optimized by the Adam optimization algorithm, perform channel pruning and INT8 quantization, dynamically adjust the learning rate through the cosine annealing strategy, and output information on the types of bougainvillea polychrome and the proportion of each color gradient region. The display module is used to convert the information on the types of bougainvillea polychrome and the proportion of each color gradient area into visual information for display. The visual information includes a gradient heat map of color space mapping, as well as a classification result map labeled with polychrome type identifiers and gradient area proportions.

8. The bougainvillea multi-colored petal classification system based on the fusion of multimodal sensors and deep learning algorithms according to claim 7, characterized in that, The system also includes a classification optimization module, which is used to dynamically adjust the saturation threshold of the bougainvillea polychrome area based on the pre-acquired latitude and longitude data of the bougainvillea plants. Specifically, this includes: dynamically adjusting the saturation threshold according to a preset latitude-threshold mapping relationship based on the latitude information in the latitude and longitude data; and triggering a corresponding optical compensation mechanism when a specific lighting condition is detected, based on real-time ambient light intensity data. The specific lighting condition includes light intensity meeting a preset range.

9. The bougainvillea multi-colored petal classification system based on the fusion of multimodal sensors and deep learning algorithms according to claim 7, characterized in that, The feature extraction module includes: The edge feature extraction unit is used to input the standardized feature image into the channel-pruned MobileNetV3 network, compress the network output layer dimension from 1280 dimensions to 512 dimensions, extract features through multi-scale convolution kernels, and use max pooling layers to reduce the dimensionality of the extracted features to obtain edge features. The gradient analysis unit is used to calculate the gradient components in the horizontal and vertical directions of the image using the Sobel operator, calculate the magnitude and orientation angle of the gradient components, perform orientation feature analysis based on the HOG feature descriptor, and generate a feature vector describing the direction of color gradient. The feature fusion unit is used to normalize the edge features and gradient feature vectors, and generate the composite feature vector by feature concatenation or weighted fusion.

10. A terminal, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the terminal to implement the bougainvillea multi-colored petal classification method as described in any one of claims 1-6.

Images