A crop nutrition index detection method based on spectral images and related devices

Abstract

The application discloses a crop nutrition index detection method and related device based on spectral images, and relates to the technical field of crop nutrition detection. The method collects an original visible light image, an original spectral image and a working distance value, and completes one-way pixel coordinate registration of the visible light image by using fixed registration parameters; based on obtained multi-scale edge attention weight and spectral quality weight, the original spectral image after normalization processing and the chroma channel of the registered visible light image are adaptively weighted and fused to generate an optimized luminance channel, and the fused image is restored to a red-green-blue color space through inverse color space transformation; the fused image is segmented by using an improved lightweight semantic segmentation model to generate a binary mask matrix to extract effective spectral information, and the average relative spectral reflectance is calculated; and a lightweight nutrition index prediction model is used to output a prediction value of the crop nutrition index. The application realizes in-situ, rapid, non-destructive and accurate detection of the crop nutrition index.

Description

Technical Field

[0001] This application relates to the field of crop nutrient detection technology, and in particular to a method and related apparatus for detecting crop nutrient indicators based on spectral images. Background Technology

[0002] Currently, agricultural development is accelerating its transformation towards large-scale, intensive, and precision agriculture. Accurate measurement of crop nutrient indicators has become a crucial link in the high-quality development of modern agriculture and the guarantee of food security. Real-time and accurate acquisition of nutrient indicators such as crop nitrogen content and relative chlorophyll content in leaves is of great significance for guiding scientific fertilization, improving crop yield and quality, and reducing agricultural non-point source pollution.

[0003] Existing crop nutrient testing methods are mainly divided into two categories. One category is laboratory physicochemical testing methods, which have high detection accuracy but are complex, time-consuming, costly, and destructive, making them difficult to meet the needs of real-time, non-destructive testing in the field. The other category is conventional spectroscopic detection technology, which relies on spectral features to achieve non-destructive testing. However, existing spectroscopic detection methods generally have the following shortcomings: First, most methods rely on a single spectral data source and do not effectively integrate visible light image information, resulting in unstable spectral feature extraction and poor anti-interference ability under complex field lighting conditions. Second, traditional spectral-visible light image fusion methods often suffer from edge misalignment, color distortion, and darkened fused images, affecting the accuracy of subsequent analysis. Third, the image registration step in existing methods mostly relies on dynamic feature point extraction and iterative fitting, which is computationally intensive, inefficient, and difficult to run in real time on portable devices. Fourth, methods for crop leaf region segmentation mostly use conventional thresholding or simple machine learning models, which lack robustness to complex field backgrounds, resulting in impure spectral sampling areas and thus affecting the accuracy of nutrient index inversion.

[0004] In summary, existing crop nutrient detection methods still have significant shortcomings in terms of image registration efficiency, multimodal image fusion quality, and the lightweight and reliability of nutrient index prediction models, making it difficult to meet the actual needs of modern agriculture for rapid, non-destructive, and accurate field detection. Summary of the Invention

[0005] The purpose of this application is to provide a method and related device for detecting crop nutrient indicators based on spectral images, which can realize in-situ, rapid, non-destructive and accurate detection of crop nutrient indicators in the field.

[0006] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for detecting crop nutrient indicators based on spectral images, including: In response to user-triggered start and acquisition commands, it acquires raw visible light images, raw spectral images, and working distance values; Based on the working distance value and the preset fixed registration parameters, the original visible light image is unidirectionally mapped to the original spectral image to generate a registered visible light image; An image fusion operation is performed on the original spectral image and the registered visible light image to generate a fused image. The image fusion operation is based on the acquired multi-scale edge attention weights and spectral quality weights. The original chroma channel of the normalized original spectral image and the original chroma channel of the registered visible light image are adaptively weighted and fused to generate a fused image with optimized luminance channel and restored to the red-green-blue color space through inverse color space transformation. The target leaf region of the fused image is segmented using a pre-defined improved lightweight semantic segmentation model to generate a binary mask matrix; Based on the binarized mask matrix, the original spectral digital quantization value corresponding to the effective pixel position is extracted from each spectral band of the acquired original spectral data. The first average original spectral digital quantization value of each spectral band is calculated, and the average relative spectral reflectance of each spectral band is calculated based on the first average original spectral digital quantization value of each spectral band. The first average original spectral digital quantization value is the average original spectral digital quantization value of the target crop area. Based on the average relative spectral reflectance of each spectral band, a preset lightweight nutrient index prediction model is used to invert nutrient indexes and output the predicted values ​​of crop nitrogen content and leaf relative chlorophyll content.

[0007] Secondly, this application provides a portable crop nutrient index detection device based on spectral images, comprising: The main control mechanism includes a main control board and a slave control board, the main control board being connected to the slave control board, and the main control board being used to execute a crop nutrient index detection method based on spectral images; A spectral acquisition mechanism includes a visible light camera and a spectral camera, which are respectively connected to the main control board. The visible light camera is used to acquire raw visible light images, and the spectral camera is used to acquire raw spectral images. An auxiliary sensing mechanism is provided, comprising a laser ranging module and a communication positioning module. The laser ranging module is connected to the slave control board, and the communication positioning module is directly connected to the main control board. The laser ranging module is used to collect working distance values ​​in real time and transmit the working distance values ​​to the main control board through the slave control board. The communication positioning module is used to acquire the positioning data of the device and realize data interaction communication. The display mechanism is connected to the main control board and is used to display the test results and the human-machine interface. The control mechanism includes physical buttons and a touch unit. The physical buttons are connected to the main control board and the slave control board respectively, and the touch unit is connected to the main control board. Both the physical buttons and the touch unit are used to receive user operation commands. The power supply mechanism is electrically connected to the main control board, the spectrum acquisition mechanism, the display mechanism, and the slave control board, and is used to provide working power.

[0008] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a method and related device for detecting crop nutrient indicators based on spectral images. By mapping the original visible light image to the original spectral image in one direction according to the working distance value and preset fixed registration parameters, a registered visible light image is generated. This solves the problem of high computational load and low efficiency caused by the reliance on dynamic feature point extraction and iterative fitting in existing methods. It achieves fast, stable, and pixel-level accurate registration based on fixed hardware structure and real-time distance compensation, significantly reducing the computing power requirements of portable devices.

[0009] A fused image is generated by performing an image fusion operation on the original spectral image and the registered visible light image. The image fusion operation is based on the acquired multi-scale edge attention weights and spectral quality weights. The chromaticity channels of the normalized original spectral image and the registered visible light image are adaptively weighted and fused to generate a fused image with optimized brightness channels. This image is then restored to the red-green-blue color space through inverse color space transformation. This solves the problems of edge misalignment, color distortion, and dark fused images in traditional spectral-visible light image fusion methods. It achieves lightweight and efficient image fusion without deep learning training, and can generate high-quality fused images with clear edges, realistic colors, and uniform brightness in real time on embedded devices.

[0010] By using a pre-defined improved lightweight semantic segmentation model to segment the target leaf region of the fused image and generate a binary mask matrix, the problem of insufficient robustness of existing segmentation methods to complex field backgrounds is solved. This achieves accurate extraction of the target leaf region and effective removal of background interference, providing a clean and reliable effective leaf region for subsequent spectral sampling.

[0011] By extracting the original spectral digital quantization value corresponding to the effective pixel position from each spectral band of the acquired original spectral data according to the binarized mask matrix, the first average original spectral digital quantization value of each spectral band is calculated, and the average relative spectral reflectance of each spectral band is calculated based on the first average original spectral digital quantization value of each spectral band. This solves the problem of impure spectral sampling area and severe interference from background noise in traditional methods, and realizes effective spectral data extraction and normalization of band by band based on mask guidance, providing high-quality spectral input for nutrient index inversion.

[0012] By using the average relative spectral reflectance of each spectral band and employing a pre-set lightweight nutrient index prediction model, nutrient index inversion is performed, outputting predicted values ​​of crop nitrogen content and relative chlorophyll content in leaves. This solves the problems of high computing power requirements, difficulty in embedded deployment, and poor prediction stability of existing nutrient index prediction models, and realizes lightweight, high-precision in-situ quantitative prediction of nutrient indexes in the field, which can output nitrogen content and relative chlorophyll content in leaves in real time. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of a process for detecting crop nutrient indicators based on spectral images, provided in Embodiment 1 of this application. Figure 2 This is a schematic flowchart of a crop nutrient index detection method based on spectral images provided in Embodiment 2 of this application; Figure 3 This is a flowchart illustrating the image registration, fusion, segmentation, and mask generation process provided in Embodiment 2 of this application. Figure 4 A flowchart for establishing the crop nutrient index detection model provided in Embodiment 2 of this application; Figure 5 This is a comparison of the segmentation effects of different image fusion methods on soybean leaves provided in Embodiment 2 of this application; Figure 6 This is a first-view overall structural diagram of a crop nutrient index detection device based on spectral images provided in Embodiment 3 of this application; Figure 7 This is a first-view internal structure diagram of a crop nutrient index detection device based on spectral images provided in Embodiment 3 of this application; Figure 8 This is a second-view overall structural diagram of a crop nutrient index detection device based on spectral images provided in Embodiment 3 of this application; Figure 9 This is a second-view internal structure diagram of a crop nutrient index detection device based on spectral images provided in Embodiment 3 of this application; Figure 10 This is a connection block diagram of a crop nutrient index detection device based on spectral images provided in Embodiment 3 of this application.

[0015] Figure label: 1-Laser ranging module, 2-Visible light camera, 3-Spectrometer camera, 4-Type-C charging port, 5-Communication positioning module, 6-Slave control board, 7-Power switch, 8-Power control board, 9-Main control board, 10-Screen expansion board, 11-Mobile industry processor interface-display serial interface display screen, 12-3000mAh lithium battery, 13-Power wake-up button, 14-Other buttons on top, 15-External antenna interface, 16-Direction buttons, 17-Acquisition button, 18-Confirm button, 19-Attitude sensor. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] Example 1 In one exemplary embodiment, such as Figure 1 As shown, a method for detecting crop nutrient indicators based on spectral images is provided. This method is executed by a master control mechanism, specifically by a master control board and a slave control board. In this embodiment, it includes steps 101 to 106. Wherein: Step 101: In response to the user-triggered start command and acquisition command, acquire the original visible light image, the original spectral image, and the working distance value.

[0019] Step 102: Based on the working distance value and the preset fixed registration parameters, the original visible light image is unidirectionally mapped to the original spectral image to generate the registered visible light image.

[0020] Step 103: Perform image fusion operation on the original spectral image and the registered visible light image. The image fusion operation is based on the acquired multi-scale edge attention weights and spectral quality weights. The original spectral brightness image (i.e., the normalized spectral brightness image) and the original chromaticity channel of the registered visible light image are adaptively weighted and fused to generate a fused image with optimized brightness channels and restored to the red-green-blue color space through inverse color space transformation.

[0021] Step 104: Use the preset improved lightweight semantic segmentation model to segment the target leaf region of the fused image and generate a binary mask matrix.

[0022] Step 105: Based on the binarization mask matrix, extract the original spectral digital quantization value corresponding to the effective pixel position from each spectral band of the acquired original spectral data, calculate the first average original spectral digital quantization value of each spectral band, and calculate the average relative spectral reflectance of each spectral band based on the first average original spectral digital quantization value of each spectral band; the first average original spectral digital quantization value is the average original spectral digital quantization value of the target crop area.

[0023] Step 106: Based on the average relative spectral reflectance of each spectral band, use the preset lightweight nutrient index prediction model to perform nutrient index inversion and output the predicted values ​​of crop nitrogen content and leaf relative chlorophyll content.

[0024] By implementing steps 101 to 106 above, this embodiment has the following beneficial effects: (1) By mapping the original visible light image to the original spectral image in one direction according to the working distance value and the preset fixed registration parameters, a registered visible light image is generated. This solves the problem of large computation and low efficiency caused by the reliance on dynamic feature point extraction and iterative fitting in the existing method. It realizes fast, stable and pixel-level accurate registration based on fixed hardware structure and real-time distance compensation, which significantly reduces the computing power requirements of portable devices.

[0025] (2) By performing image fusion operation on the original spectral image and the registered visible light image, a fused image is generated. The image fusion operation is based on the acquired multi-scale edge attention weight and spectral quality weight. The normalized original spectral image and the registered visible light image are adaptively weighted and fused to generate a fused image with optimized brightness channel and restored to red-green-blue color space by inverse color space transformation. This solves the problems of edge misalignment, color distortion and dark fused image in traditional spectral-visible light image fusion methods. It realizes image fusion that does not require deep learning training, is lightweight and efficient, and can generate high-quality fused images with clear edges, real colors and uniform brightness in real time on the embedded side.

[0026] (3) By using the preset improved lightweight semantic segmentation model to segment the target leaf region of the fused image and generate a binary mask matrix, the problem of insufficient robustness of the existing segmentation method to complex field backgrounds is solved. The accurate extraction of the target leaf region and the effective removal of background interference are realized, providing a clean and reliable effective leaf region for subsequent spectral sampling.

[0027] (4) By extracting the original spectral digital quantization value corresponding to the effective pixel position from each spectral band of the acquired original spectral data according to the binarization mask matrix, the first average original spectral digital quantization value of each spectral band is calculated, and the average relative spectral reflectance of each spectral band is calculated based on the first average original spectral digital quantization value of each spectral band. This solves the problem of impure spectral sampling area and severe interference from background noise in traditional methods, and realizes effective spectral data extraction and normalization based on mask guidance for each band, providing high-quality spectral input for nutritional index inversion.

[0028] (5) By using the preset lightweight nutrient index prediction model based on the average relative spectral reflectance of each spectral band, the nutrient index is inverted and the predicted values ​​of crop nitrogen content and leaf relative chlorophyll content are output. This solves the problems of high computing power requirement, difficulty in embedded deployment and poor prediction stability of existing nutrient index prediction models. It realizes lightweight and high-precision in-situ quantitative prediction of nutrient index in the field and can output nitrogen content and leaf relative chlorophyll content in real time.

[0029] In summary, through the synergistic effect of the above steps, this application effectively overcomes the shortcomings of existing crop nutrient detection methods in terms of image registration efficiency, multimodal image fusion quality, leaf segmentation robustness, spectral sampling purity, and the lightweight and reliability of prediction models. It achieves in-situ, rapid, non-destructive, and accurate detection of crop nutrient indicators in the field, fully meeting the practical application needs of modern agriculture for portable detection devices.

[0030] Further, the fixed registration parameters in step 102 include the fixed lateral offset and the fixed longitudinal offset of the target center between the visible light camera and the spectral camera, as well as the lateral pixel mapping scaling factor and the longitudinal pixel mapping scaling factor that are dynamically updated with the working distance; based on the working distance value and the preset fixed registration parameters, the original visible light image is unidirectionally mapped to the original spectral image to generate the registered visible light image, specifically including: Step 1021: Construct a third-order registration matrix with distance compensation based on the working distance value, the horizontal pixel mapping scaling factor, the vertical pixel mapping scaling factor, the horizontal fixed offset of the target center, and the vertical fixed offset of the target center.

[0031] Step 1022: Using a third-order registration matrix, perform homogeneous coordinate transformation on the coordinates of each pixel in the visible light image to generate the registered visible light image.

[0032] Furthermore, step 103 involves performing an image fusion operation on the original spectral image and the registered visible light image to generate a fused image, specifically including: Step 1031: Expand the original single-channel spectral image into a three-channel spectral image by channel copying.

[0033] Step 1032: Using a preset multi-scale Laplacian convolution kernel, edge features are extracted from the three-channel spectral image and the registered visible light image respectively, to obtain spectral edge feature maps and visible light edge feature maps at each scale; wherein, the weights of each Laplacian convolution kernel are fixed values ​​and do not require training or updating.

[0034] Step 1033: Calculate the edge difference between the spectral edge feature map and the visible light edge feature map at each scale to obtain the edge difference map at each scale, and perform multi-scale feature fusion on the edge difference maps at all scales to generate a fused edge difference map.

[0035] Step 1034: Calculate pixel-by-pixel normalized similarity based on the fused edge difference map to generate an edge similarity map. Perform nonlinear mapping and linear transformation processing on the edge similarity map to generate an edge attention weight map.

[0036] Step 1035: Convert the registered visible light image from the original color space to a luminance and chrominance separated color space to separate the original luminance channel and the original chrominance channel.

[0037] Step 1036: Obtain the pixel value distribution of the original spectral image, and perform percentile normalization on the original spectral image based on the statistical characteristics of the pixel value distribution to generate a normalized spectral brightness image, which is the original spectral image after normalization.

[0038] Step 1037: Perform local texture sharpness assessment on the original spectral image to generate a spectral quality weight map; the spectral quality weight map is used to characterize the imaging quality of the original spectral image at each pixel location.

[0039] Step 1038: Perform linear weighted fusion processing on the edge attention weight map and the spectral quality weight map to obtain the weighted fusion result, and impose numerical range constraints on the weighted fusion result to generate a comprehensive adaptive weight map.

[0040] Step 1039: Perform pixel-by-pixel weighted calculations on the normalized spectral brightness image and the comprehensive weight map to generate the optimized brightness channel.

[0041] Step 10310: Replace the original luminance channel of the registered visible light image with the optimized luminance channel, and combine it with the retained original chrominance channel to restore it into a fused image of red, green and blue color space through inverse color space transformation.

[0042] Further, in step 105, based on the binarized mask matrix, the original spectral digital quantization value corresponding to the effective pixel position is extracted from each spectral band of the acquired original spectral data, the first average original spectral digital quantization value of each spectral band is calculated, and the average relative spectral reflectance of each spectral band is calculated based on the first average original spectral digital quantization value of each spectral band, specifically including: For each spectral band of the acquired raw spectral data, a relative spectral reflectance acquisition operation is performed to obtain the average relative spectral reflectance of each spectral band; specifically, the relative spectral reflectance acquisition operation is as follows: Step 1051: Based on the binarized mask matrix, the original spectral digital quantization values ​​corresponding to the positions with a mask value of 1 are accumulated to obtain the original spectral digital quantization accumulated value. The positions with a mask value of 1 are taken as valid pixels, and the number of valid pixels is counted. Based on the original spectral digital quantization accumulated value and the number of valid pixels, the first average original spectral digital quantization value is calculated.

[0043] Step 1052: Obtain whiteboard data; the whiteboard data includes the second average original spectral digital quantization value for each spectral band, and the second average original spectral digital quantization value is the average original spectral digital quantization value of the calibration whiteboard.

[0044] Step 1053: Dark current correction is performed on the first average original spectral digital quantization value and the second average original spectral digital quantization value of the corresponding spectral band to obtain the corrected first average original spectral digital quantization value and the corrected second average original spectral digital quantization value. Based on the corrected first average original spectral digital quantization value and the corrected second average original spectral digital quantization value, the average relative spectral reflectance is calculated.

[0045] Further, in step 106, based on the average relative spectral reflectance of each spectral band, a preset lightweight nutrient index prediction model is used to invert nutrient indices, outputting predicted values ​​for crop nitrogen content and relative chlorophyll content in leaves, specifically including: Step 1061: Based on the average relative spectral reflectance of each spectral band and the actual physical reflectance of the calibration whiteboard for the corresponding spectral band, calculate the average absolute spectral reflectance of each spectral band.

[0046] Step 1062: The average absolute spectral reflectance of each spectral band is smoothed using a Savitzky-Gore smoothing filter with a window length of 7 to obtain the smoothed spectral data for each spectral band.

[0047] Step 1063: Perform a first-order derivative transformation on the smoothed spectral data for each spectral band to obtain the transformed spectral data for each spectral band.

[0048] Step 1064: Perform standard normal transformation on the spectral data after transformation of each spectral band to obtain the spectral characteristics of each spectral band.

[0049] Step 1065: Input the spectral features of each spectral band into the lightweight nutrient index prediction model, and output the predicted values ​​of crop nitrogen content and leaf relative chlorophyll content; the lightweight nutrient index prediction model is a model constructed based on the XGBoost regression algorithm and obtained by Bayesian optimization of hyperparameters.

[0050] Example 2 In another exemplary embodiment of this application, a portable crop nutrient index detection method based on spectral images is provided. This method is applied to a portable crop nutrient index detection device, which includes a main control mechanism, a spectral acquisition mechanism, an auxiliary sensing mechanism, a display mechanism, a control mechanism, and a power supply mechanism. The method in this embodiment is mainly executed by the main control mechanism in the device. The following description, in conjunction with... Figure 2 The flowchart shown will be explained in detail.

[0051] Step S1: Start the detection device: In response to a user-triggered startup command (such as pressing the power button), a signal is transmitted to the main control mechanism, triggering system initialization and displaying the startup interface on the Mobile Industry Processor Interface-Display Serial Interface (MIPI-DSI) 11. Subsequently, the crop nutrient index spectral detection software is automatically opened, and the data acquisition interface in the software is displayed by default.

[0052] Step S2: Activate the auxiliary sensing mechanism: In the data acquisition interface, users can activate the ranging, attitude sensing, and positioning functions by clicking the corresponding on-screen display button or physical button. After activation, the interface will display the returned distance, angle, and location information in real time.

[0053] Step S3: Activate the spectral acquisition mechanism: Users can simultaneously activate the spectral camera 3 and the visible light camera 2 via the camera switch or corresponding physical button on the MIPI-DSI display 11. After activation, the display status of the two cameras can be viewed in the corresponding window. The visible light camera 2 is used to acquire raw visible light images (three-channel red, green, and blue images, i.e., RGB images), while the spectral camera 3 is used to acquire raw spectral images (single-channel full-band spectral data) and raw spectral digital quantization values ​​(hereinafter referred to as DN values).

[0054] Step S4: Adjust the detection device: Align the acquisition plane of the device with the blade to be tested, and adjust the distance between the device and the blade and the tilt of the device to a reasonable range by observing the distance and angle values ​​displayed in real time on the interface. To obtain better detection results, the target blade should be kept as flat and unobstructed as possible to ensure that the image of the blade acquired by the visible light camera 2 is clear, complete, and covers the entire target area.

[0055] Step S5: Adjust the exposure time: Adjust the exposure time of the spectral camera 3 using the exposure time adjustment slider until the spectral camera 3 can clearly display the original spectral image without overexposure.

[0056] Step S6: Collect data: Once the image of the detected target is confirmed to be clear, complete, and in a suitable position, the user can press the acquisition button in the software interface or the corresponding physical acquisition button 17 to simultaneously save the current visible light image, the current spectral image, and the corresponding spectral data, as well as various auxiliary sensing data (including working distance value, attitude data, and positioning information), providing data support for subsequent analysis. In this step, in response to the acquisition command triggered by the user, the original visible light image, the original spectral image, and the working distance value are acquired.

[0057] Step S7: Switch to the data analysis interface and select the target image: After data acquisition is complete, you can switch to the data analysis interface. The default image displayed in the data analysis interface is the most recently acquired image. Users can use the "Previous Page" and "Next Page" buttons on the touch screen, or press the directional button 16 in combination with the confirmation button 18 to navigate to the target image to be read.

[0058] Step S8: Image registration, fusion, segmentation, and mask generation: After adjusting to the target image, the user touches the recognition button on the interface, or presses the directional button 16 to switch to the recognition function and then presses the confirmation button 18. This automatically registers and fuses the visible light image into the spectral image, identifies the crop species in the fused image, and automatically segments the target area, generating a mask for the corresponding target region. For example... Figure 3 As shown, this step specifically includes S81 to S84.

[0059] S81: Image Registration Image registration is performed on the acquired visible light and spectral images, a step completed on the detection device. This application adopts a modular structural design. The visible light camera 2 and the spectral camera 3 are fixedly installed within the device with precise calibration before leaving the factory. The relative installation position, lens optical axis parallelism, sensor target surface offset, lens focal length, sensor pixel size, field of view, and other inherent hardware parameters are all constant values. During operation, the two cameras do not have any internal relative displacement, angular deflection, or structural deformation. The relative geometric relationship between the two cameras remains constant, ensuring the stability of the image mapping relationship at the hardware level.

[0060] Because the visible light camera 2 and the spectral camera 3 have a fixed front-to-back mounting distance (inconsistent optical axes), changes in the working distance between the crop leaf being measured and the detection device directly affect the pixel mapping relationship. Therefore, this application uses a built-in laser ranging module 1 (TOF ranging module) to acquire the distance to the target being measured (i.e., the working distance value Z) in real time, and, combined with a fixed hardware structure, constructs a distance-compensated unidirectional homogeneous coordinate transformation model from a visible light image to a spectral image. The coordinates of any pixel in the visible light image are defined as (u... v ,v v The precise aligned pixel coordinates of this pixel on the spectral image are (u s ,v s The mapping relationship between the two satisfies formula (1): (1); In the formula, M(Z) is a 3×3 registration matrix with distance compensation, and its specific expression is shown in formula (2): (2); Where Z is the working distance measured in real time by the TOF ranging module, and k x (Z), k y (Z) represent the horizontal pixel mapping scaling factor and the vertical pixel mapping scaling factor, which are dynamically updated with the working distance, respectively. , These are the fixed lateral and longitudinal offsets of the target center for both cameras. These fixed parameters are calibrated at the factory, and the scaling factor is automatically calculated from the real-time distance.

[0061] This registration scheme undergoes a one-time precise calibration and error calibration using a standard calibration board before the device leaves the factory. After successful calibration, the parameters are permanently stored in the device's built-in storage unit as proprietary factory-fixed parameters. Compared to traditional dynamic registration methods, this method eliminates the need for feature point extraction and iterative fitting, eliminating parallax errors caused by front-to-back camera mounting solely through a fixed hardware structure and real-time distance compensation. This ensures pixel-level accurate registration at different working distances, significantly reducing the device's computational load and power requirements. It also avoids error accumulation associated with dynamic registration, balancing registration efficiency and pixel-level alignment accuracy. This results in greater stability and better suitability for the practical needs of rapid, non-destructive, and continuous crop monitoring in the field.

[0062] S82: Image Fusion Image fusion is performed on the registered visible light image and the original spectral image, a step completed on the detection device. The original spectral image used here is full-band spectral data acquired by the spectral camera 3, directly derived from the camera's photoelectric conversion output. Only the spectral camera 3 performs the built-in basic correction, without radiometric calibration, reflectivity normalization, or external secondary post-processing. It completely preserves the spatial structure information, showing only overall brightness fluctuations under changes in illumination, and possesses strong spatial structure robustness.

[0063] Nutrient index retrieval relies on the spectral characteristics of leaf regions and is highly sensitive to regional purity, pixel positioning accuracy, and result stability. A single visible light image is susceptible to interference from illumination and shadows, leading to segmentation distortion. While a single raw spectral image exhibits strong resistance to light interference, it lacks visible light color information, making it prone to missed or incorrect segmentation. Using either alone results in spectral sampling errors, directly reducing the accuracy of nutrient index retrieval. Multimodal image fusion combines the high-precision positioning advantage of visible light with the strong anti-interference capability of the raw spectral image. It corrects brightness distortion in the visible light image based on the data characteristics after spectral brightness normalization, while preserving visible light color information. This effectively highlights normally illuminated leaf regions and weakens interference from shadowed areas, obtaining pure and reliable effective leaf regions. This ensures the accuracy of spectral sampling data and achieves high-precision and high-stability nutrient index retrieval.

[0064] To address the technical challenges in fusing original spectral images with visible light images, such as edge misalignment and ghosting, suppression of visible light colors by the original spectral information, misclassification due to invalid leaves in shadow areas, and the high computational demands of existing fusion networks making them unsuitable for embedded low-computing-power platforms, this application proposes a lightweight image fusion method based on multi-scale edge attention, spectral quality perception, and a brightness-chromaticity separation color space (i.e., the YCbCr color space, where Y is the brightness channel, Cb is the blue chromaticity channel, and Cr is the red chromaticity channel). This method requires no training or deep learning networks, has minimal computational cost, and extremely fast inference speed, allowing it to run in real-time on embedded hardware. While ensuring clear edges and eliminating ghosting in the fused image, it fully preserves the true colors of visible light and the structural information of the spectral image, and optimizes the brightness distribution in shadow areas, thus resolving issues of missed and incorrect segmentation.

[0065] The fusion method designed in this application embodiment comprises five core components: an edge attention module, a YCbCr color space conversion module, a spectral brightness normalization module, a spectral quality perception module, and a weighted fusion module. To adapt to the three-channel input requirements of the edge attention module, and considering that the original spectral image is single-channel raw acquisition data, this application embodiment expands it to a three-channel format through channel copying without any numerical transformation, thus fully preserving the original spectral information and image details and textures. The following sections will elaborate on each of the five core modules.

[0066] S82.1: Edge Attention Module The edge attention module is divided into three interconnected core parts.

[0067] First, the first part of the edge attention module is executed: multi-scale edge feature extraction. Three sets of Laplacian convolutional kernels with fixed parameters at scales of 3, 5, and 7 are used in parallel to extract edge features, adapting to three scale features: fine texture of small lesions, medium leaf texture, and large outline of the entire leaf. The weights of the basic 3×3 convolutional kernels are fixed. The 5 and 7 large-scale convolutional kernels are obtained by scaling and normalizing the basic 3×3 kernels through bilinear interpolation. All convolutional kernel weights are fixed and do not require training updates. The weights of the basic 3×3 convolutional kernels are set as in formula (3): (3); In the formula, K is a 3×3 Laplacian edge detection convolution kernel used to extract high-frequency edge features of the image, with fixed weights that do not require training or updating. The 5×5 and 7×7 scale convolution kernels are obtained by interpolating, scaling, and normalizing K to ensure uniform edge response amplitude at different scales. The general calculation operation for edge extraction of a single image is formula (4): (4); In the formula, This represents a two-dimensional convolution operation; I is the three-channel input image; E is the single-channel edge feature map. The original three-channel spectral images I after channel expansion are then processed. gray After registration, the three-channel visible light image I vis Substituting into formula (4), for each set of Laplacian convolution kernels at scales 3, 5, and 7, a set of spectral edge feature maps and visible light edge feature maps are generated. The edge difference between two images is calculated scale by scale. The calculation principle is as follows: (5); In formula (5), This is the edge difference map at the k-th scale, representing the degree of edge misalignment in the region at this scale. , These are the spectral edge feature map and the visible light edge feature map at the k-th scale, respectively; k takes values ​​of 3, 5, and 7. The three sets of scale edge difference maps are concatenated along the channel dimension, and multi-scale misalignment features are adaptively fused using a 1×1 convolution to obtain a unified fused edge difference map. (6); In formula (6), [ This refers to channel splicing operations; It is a 1×1 convolution with only a very small number of optional fine-tuning parameters, used to adaptively fuse misalignment information at different scales.

[0068] Next, the second part of the edge attention module is executed: edge similarity calculation. Based on the fused edge difference map, pixel-by-pixel normalized similarity is calculated to eliminate interference from image brightness differences, resulting in an edge similarity map. The specific formula is as follows: (7); In formula (7), To merge the edge difference map; E sim For the edge similarity map, max(E) diff,fused ) represents the global maximum value of the merged edge difference map, 10 -8 This is a minimum value to prevent calculation errors caused by a denominator of zero.

[0069] Finally, the third part of the edge attention module is executed: edge attention weight generation. Using exponential nonlinear operations, the edge similarity map is mapped pixel-by-pixel to the 0-1 interval, and then constrained to the 0.7-1.0 interval through linear transformation, generating an edge attention weight map. The closer the weight value is to 1, the higher the edge alignment of the corresponding region; the closer the weight value is to 0.7, the more obvious the edge misalignment of the corresponding region. This achieves the effect of strengthening aligned regions and suppressing misaligned regions. The formula for generating edge attention weights is: (8); In the formula, W edgeis a single-channel edge attention weight map used to differentiate weights for different regions of the image; e is a natural constant, which maps the value to the 0-1 range through exponential nonlinear operations to achieve reasonable constraints on the weights; 0.7 is the lower limit constraint value of the weights, used to ensure that even in areas with severe edge misalignment, the spectral brightness information is still retained at least 70%, avoiding black holes or brightness collapse in the fused image; 0.3 is the weight adjustment range, used to control the rate of change of the weights with the edge alignment.

[0070] This module outputs a single-channel edge attention weight map. This characterizes the edge matching degree between the two input images in each region of the image, providing a basis for registration deviation constraints for subsequent dual-weight fusion.

[0071] S82.2: YCbCr color space conversion module: To ensure the fused image has realistic, natural, and undistorted colors, the visible light image is converted from the RGB color space to the YCbCr color space. The luminance channel Y and the chrominance channels Cb and Cr are separated. The chrominance channels Cb and Cr retain the complete visible light color information without any modification. Only the luminance channel Y is optimized and replaced. This maintains color stability, prevents color cast, and avoids oversaturation during the fusion process, fundamentally solving the problem of spectral information suppressing visible light colors. The RGB to YCbCr conversion formula is as follows: (9); (10); (11); In the formula, R, G, and B are the pixel values ​​of the red, green, and blue channels of the registered visible light image, respectively; Y is the pixel value of the luminance channel of the visible light image, representing the brightness of the image; Cb is the pixel value of the chrominance channel, representing the blue-yellow color deviation of the image; and Cr is also the pixel value of the chrominance channel, representing the red-purple color deviation of the image.

[0072] S82.3: Spectral Brightness Normalization Module: This module independently performs brightness preprocessing of the original spectral image. It overcomes the shortcomings of traditional global extremum normalization, which is susceptible to extreme noise interference, and adopts a percentile robust normalization method to remove the top and bottom 2% of abnormal extreme pixels in the image. This eliminates brightness interference caused by sensor dark current and single-point overexposure defects. The relevant calculation formulas are as follows: (12); In the formula, I spec These are the pixel values ​​of the raw spectral image in a single channel. (I spec () represents the 2nd percentile pixel value of the raw spectral image from a single channel; (I spec() represents the 98th percentile pixel value of the raw single-channel spectral image; This is a temporary brightness map after percentile normalization.

[0073] S82.4: Spectral quality sensing module: This module performs a global pixel quality assessment on the single-channel raw spectral image independently, without relying on visible light image information. It judges the imaging quality of each region of the spectral image from two dimensions: local contrast and local gradient sharpness. It distinguishes between high signal-to-noise ratio (SNR) sharp leaf regions and low SNR noisy and blurry regions, thus compensating for the edge attention module's inability to identify inherent imaging defects in the spectrum itself. The specific calculation process is as follows: Step 1: Calculation of local mean and local contrast. A 5×5 moving average window is used to perform local feature statistics. The calculation formula is as follows: (13); (14); In the formula, The original spectral image in pixel coordinates The pixel value at that location; [i,j] is the mean of all 25 pixels in a 5×5 neighborhood centered at [i,j]; C[i,j] is the local contrast pixel value at the same coordinates. The local variance is obtained by first calculating the squared difference between the pixel value in the neighborhood and the local mean, and then taking the neighborhood average. 10 -8 It is a very small constant to avoid numerical calculation errors caused by a value of 0 within the square root; the overall square root is used to obtain the contrast that characterizes the degree of fluctuation in window texture.

[0074] Step 2: Local gradient calculation. Based on the Sobel operator, the horizontal and vertical gradients of the image are extracted. The magnitude of the local gradient is calculated to represent the sharpness of the region. The calculation formula is as follows: (15); (16); (17); In the formula, , ( These are the horizontal and vertical Sobel gradient convolution operations, respectively, used to detect texture changes in the horizontal and vertical directions of an image. , These are pixel-by-pixel horizontal and vertical gradient feature maps, reflecting the degree of local edge and texture changes in the left-right and up-down directions of the image; The gradient magnitude feature map is synthesized pixel by pixel to represent the overall texture clarity within a 3×3 neighborhood of a single pixel. The larger the magnitude, the clearer the spectral texture of the corresponding area and the higher the imaging quality.

[0075] Step 3: Joint scoring of two indicators and weight smoothing constraints. Pixel-level quality is assessed by combining contrast and gradient sharpness, and weights are smoothed to eliminate pixel abrupt changes. Finally, the weight range is constrained, calculated using the following formula: (18); (19); (20); In the formula, [i,j] represents the initial spectral quality score pixel value at coordinate [i,j]; C[i,j], [i,j] are taken from the values ​​of the contrast feature map and the gradient magnitude feature map at the corresponding pixel coordinates, respectively; 0.05 is the local contrast qualification threshold; 0.02 is the gradient sharpness qualification threshold; 20 and 30 are the amplification coefficients of the two indicators, which enhance the distinction between high and low quality; [i,j] are the image pixel coordinates; The initial quality scoring image is determined by taking the pixel values ​​at each position within a 3×3 neighborhood centered at [i,j]. The score of a single pixel at coordinate [i,j] after smoothing the mean of the neighborhood of a 3×3 window is used to eliminate abrupt noise in a single pixel. The complete and smoothed quality score chart; The final spectral quality weight map is shown; 0.3 is the lower limit of the quality weight, where at least 30% of the spectral information is retained in the noisy region; 0.7 is the dynamic adjustment range of the weight.

[0076] S82.5: Dual-weighted adaptive weighted fusion module: This module takes the previously output edge attention weights, spectral quality weights, and normalized spectral brightness maps, and completes three steps: dual-channel weight fusion, adaptive weighted optimization of spectral brightness, and inverse color space transformation to restore the RGB image.

[0077] Step 1: Dual-channel weighted fusion. An adaptive weighted fusion is performed by combining edge alignment accuracy weights and spectral imaging quality weights. This prioritizes ensuring image edge registration accuracy, then suppresses spectral noise interference, and simultaneously imposes a lower limit constraint on the overall weights to prevent complete loss of spectral information when there is large-area misalignment at the edges. This step consists of two consecutive calculation stages: linear weighting and weight interval constraint, corresponding to formulas (21) and (22). (twenty one); (twenty two); In the formula, This is the original comprehensive weight map obtained by linear fusion with two weights, i.e., the weighted fusion result; This is the final integrated adaptive weight map after interval constraints. Formula (21) completes the linear weighted fusion of edge attention weight and spectral quality weight. 0.6 and 0.4 are the allocation coefficients of edge alignment weight and spectral quality weight, respectively. Formula (22) performs numerical constraint processing on the original integrated weight, limiting the weight to the range of 0.4~1.0 to avoid extreme values ​​of weight leading to brightness collapse and loss of local information in the fused image. The two variables are calculated sequentially and perform their respective functions. There are no repeated definitions of variables, which completely matches the serial calculation logic of the code.

[0078] Step 2: Adaptive Weighted Optimization of Spectral Brightness. The normalized spectral brightness map and the comprehensive weight map are multiplied pixel-by-pixel. The calculation formula is as follows: (twenty three); In the formula, Y final represents the optimized luminance channel pixel value; ⊙ represents the pixel-by-pixel multiplication operation, used to apply edge attention weights to the spectral luminance image, thereby strengthening aligned regions and weakening misaligned regions.

[0079] Step 3: Inverse color space transformation to restore the RGB image. This uses a newly optimized luminance channel Y. final The visible light luminance channel Y in the YCbCr color space is directly replaced, while the visible light Cb and Cr chrominance channels remain unchanged. Finally, the final three-channel fused RGB image is obtained by inverse color transformation. The formula is as follows: (twenty four); (25); (26); In the formula, Y final The values ​​are: 1. Optimized luminance channel pixel values; Cr and Cb are the chrominance channel pixel values ​​after the registered visible light image is converted to the YCbCr space; R, G, and B are the red, green, and blue channel pixel values ​​of the final fused image. The final output is a high-quality fused RGB image.

[0080] In summary, this application sequentially suppresses multi-scale registration misalignment artifacts through a multi-scale edge attention module, achieves complete color fidelity through a YCbCr color space conversion module, suppresses spectral extreme noise through an independent spectral brightness normalization module, evaluates spectral imaging sharpness through an independent spectral quality perception module, and finally achieves dual-weight adaptive pixel-level fusion through a weighted fusion module. The entire algorithm has no deep learning training process, contains only a very small number of optional fine-tuning parameters, has extremely low computational overhead, and is suitable for real-time deployment on embedded low-computing-power detection devices. It can simultaneously solve defects such as edge ghosting, color distortion, spectral noise, brightness distortion, and invalid leaf segmentation misjudgment in shadow areas of heterogeneous images, outputting high-quality leaf fusion images, effectively improving the accuracy of subsequent leaf segmentation and the stability of crop nutrient index inversion.

[0081] S83: Image Segmentation Image segmentation of the target region is performed on the detection device. To address the issue of complex background interference in field-collected images, such as soil, weeds, straw, shadows, and residual branches, which easily leads to high noise in spectral data and increased errors in nutrient index detection, an improved lightweight YOLOv5-seg semantic segmentation model is used to segment the target in the fused image obtained from S82. To achieve efficient model training and terminal deployment, a complete data annotation, model training, model transformation, and model embedding process is constructed, with the specific steps as follows.

[0082] S83.1: Data Labeling: Visible light images and raw spectral images of crop functional leaves under different lighting conditions, growth stages, and background environments were collected to construct a segmentation dataset that closely resembles the actual field scene. The ISAT intelligent automatic annotation tool was used to perform pixel-level semi-automatic annotation on the collected raw visible light images and raw spectral images, generating JSON format annotation files. Using the built-in format conversion function of the ISAT tool, the JSON format annotation files were converted into TXT format annotation files adapted to the YOLOv5-seg model with one click, providing high-quality annotation data for subsequent model training.

[0083] S83.2: Model Training: The labeled, one-to-one correspondence of raw visible light images and raw spectral image data were uniformly divided into training and validation sets in an 8:2 ratio. The training set was used for parameter optimization of the improved lightweight YOLOv5-seg model, while the validation set was used to evaluate the model's segmentation accuracy and generalization performance. The default data preprocessing method of the YOLOv5-seg framework was adopted, and basic normalization processing was performed on the training set images to ensure a consistent input format for model training.

[0084] Model training is completed using computer hardware, employing the native YOLOv5-seg training process and default parameter configuration. Multiple rounds of iterative training are conducted with the model's built-in segmentation loss function as the optimization objective. After each round of training, the model's segmentation accuracy is evaluated using a validation set. Finally, the model weight file with the lowest validation loss and the most stable segmentation performance is retained.

[0085] To adapt to embedded deployment and improve quantization compatibility and inference speed, this application makes targeted improvements to the standard YOLOv5-seg model. The core improvements are as follows: First, the model output nodes are optimized, and redundant post-processing modules that are not conducive to quantization are eliminated to improve the model's quantization adaptability. Second, the original sigmoid weighted linear unit (SiLU) activation layer is replaced with a rectified linear unit (ReLU) activation layer to accelerate the model inference speed while ensuring segmentation accuracy and meeting the needs of real-time field detection. Third, a dedicated export parameter is added to directly output a model in the Open Neural Network Exchange (ONNX) general format that conforms to the quantization specifications of neural network inference engines (such as Rockchip Neural Network, RKNN), facilitating subsequent terminal deployment and fully meeting the low computing power and fast inference requirements of portable detection devices.

[0086] S83.3: Model Transformation: The trained, optimal YOLOv5-seg model, after exporting its code and using pre-defined export parameters, is exported to the ONNX universal format. This format enables model sharing and conversion between different deep learning frameworks, and the exported model structure and operators are compatible with the conversion and quantization requirements of the RKNN Toolkit, avoiding accuracy loss and node incompatibility issues in conventional conversions. Next, the ONNX format model is converted to RKNN format using the RKNN quantization tool, and INT8 quantization (i.e., 8-bit integer quantization) is performed simultaneously, compressing the model size, reducing computational overhead, and improving inference speed on embedded devices.

[0087] S83.4: Model Embedding: The quantized RKNN format improved YOLOv5-seg model was adapted and compiled using rknn_model_zoo-1.6.0 for inference engineering, and then solidified into the main control board 9, enabling the deployment of the portable detection device. During actual detection operations, the operator presses the device's identification button, and the device can call upon its built-in registration and fusion algorithm and YOLOv5-seg crop segmentation model to quickly complete the entire process of image registration, fusion, and segmentation.

[0088] S84: Generate target mask: Based on the target segmentation results from S83, pixel-level masks for the corresponding target detection regions are automatically generated to achieve hard separation between the target and background regions. Mask generation follows a binarization rule: pixels in the segmented and identified crop target regions are assigned a value of 1, marking them as valid detection regions; pixels in background regions such as soil and weeds are assigned a value of 0, marking them as invalid interference regions. This generates a binary mask matrix M that satisfies the following: (27); Where (i,j) are the image pixel coordinates, and M(i,j) is the pixel value corresponding to the binary mask matrix; the target crop region and background region are determined by the target segmentation results of S83. The generated pixel-level mask can accurately define the effective detection range. Subsequently, only the spectral data of the mask-marked region is extracted for nutrient index analysis, further reducing background interference and improving the accuracy and reliability of crop nutrient index detection.

[0089] Step S9: Calculation of relative spectral reflectance: Pressing the read button automatically invokes the spectral camera software package. Based on the acquired raw spectral data, the built-in whiteboard data, and the target area mask generated by identification, it generates the relative spectral reflectance of the current target area and saves it as a corresponding text file. This achieves the numerical conversion from raw spectral data without physical meaning to relative spectral reflectance with physical meaning. This step specifically includes S91 and S92.

[0090] S91: Extraction of effective spectral data: After segmenting the target crop region, for each spectral band, the system performs pixel-by-pixel judgment based on the binary mask matrix. If the current position in the mask matrix is ​​a valid identifier (1), the original spectral DN value corresponding to that position is included in the calculation; if the current position in the mask matrix is ​​an invalid identifier (0), the data at that position is discarded and not included in the calculation. The sum of all valid DN values ​​included in the calculation is divided by the number of valid pixels to obtain the average original spectral digital quantization value DN of the target crop region under that band. crop (λ). Simultaneously, using the same method, based on the built-in whiteboard area mask and the original whiteboard spectral data, the average original spectral digital quantization value DN for the corresponding band of the calibrated whiteboard is calculated. white (λ), and introduce the camera dark current offset value DN. dark (λ) is used to eliminate sensor low-temperature drift noise. Then, the obtained DN... crop (λ), DN white (λ) and DN dark (λ) The average relative spectral reflectance of the target region is calculated band by band using the following formula: (28); In the above formula, λ represents the spectral band, and R...rcl (λ) represents the average relative spectral reflectance of the target crop area in the λ-th band, DN crop (λ) represents the average original spectral digital quantization value of the target crop region, DN white (λ) represents the average original spectral DN value for the corresponding band of the 80% calibration whiteboard, where DN is... dark (λ) represents the dark current correction value for the corresponding band of the camera. The numerical conversion from the original DN value to the physically meaningful relative spectral reflectance can be completed by formula (28), eliminating the influence of system factors such as light intensity, camera exposure and integration time, providing a unified benchmark for subsequent absolute reflectance calibration and model input, and ensuring that the data format meets the requirements of subsequent spectral preprocessing, feature engineering and model input, thus ensuring the consistency of data processing logic.

[0091] S92: Structured storage: To facilitate rapid access, data traceability, and repeated verification by the embedded platform, the system automatically saves the calculated average relative spectral reflectance of each band to a designated text file in a standardized format. The file naming follows sample identification rules, ensuring unique identification and categorized storage of individual sample data and preventing data confusion and loss. This text file can be directly read and used by the upper-level SPAD and nitrogen content prediction modules without additional format conversion, simplifying the data flow process at the edge and improving real-time field monitoring efficiency.

[0092] Step S10: Nutritional index inversion: Pressing the "Calculate Indicators" button automatically uses the data read from the current image through steps S8 and S9 as input, executes the pre-deployed data preprocessing logic, generates a fixed feature set, and calls a lightweight prediction model to complete the inversion of nutrient indicators for the current target crop area. The entire process is completed on the device itself. Specifically, batch preprocessing of spectral data, multi-dimensional feature engineering, outlier removal rules, feature optimization methods, and the training and validation of the prediction model are all completed offline on the computer. After training, testing, and validating the model using a large amount of field sample data, the lightweight prediction model, which incorporates the optimal processing logic, optimized feature set, and model parameters, is deployed to the device's main control board 9 to achieve real-time detection of single samples on the device. Figure 4 As shown, this step specifically includes S101 to S105.

[0093] S101: Spectral data preprocessing: This step employs the same logic on both the computer and the device side. The computer side provides batch preprocessed data for training the model, while the device side performs preprocessing only on single-sample data. The preprocessing method in this application adopts a multi-level progressive preprocessing process, aiming to eliminate systematic errors and random noise, restore the true spectral characteristics, and optimize data quality for the stored relative spectral reflectance data.

[0094] First, absolute reflectance calibration is performed. The dimensionless average relative spectral reflectance of each band in the target crop area is multiplied band by band by band by the actual physical reflectance of each band on the 80% standard white board to convert it into a physically meaningful average absolute spectral reflectance. The conversion formula is as follows: (29); In formula (29), λ is the spectral band, and R abs (λ) represents the average absolute spectral reflectance of the target crop area in the λ-th band, R rcl (λ) represents the average relative spectral reflectance of the target crop area in the λ-th band, R white (λ) represents the actual physical reflectance of the λ-band of an 80% standard whiteboard.

[0095] Subsequently, a Savitzky-Golay (SG) smoothing filter with a window length of 7 was used to smooth the absolute reflectance curve, filtering out high-frequency random noise while preserving the original spectral characteristics. Then, a first-order derivative transformation was performed on the smoothed data to highlight the peak and valley features of the spectral curve and amplify the spectral differences between different samples. Finally, Standard Normal Variance (SNV) was used for baseline correction to eliminate baseline offset errors caused by measurement angle, leaf size, and uneven illumination, thus improving data stability. The formula for Standard Normal Variance is: (30); In formula (30), The value of the λ-th band is the spectral value after standard normal transformation. The absolute spectral reflectance after first-order derivative transformation. The mean of the first derivative spectral sequence is given. is the standard deviation of the first derivative spectral sequence.

[0096] S102: Establishing Multidimensional Feature Engineering: This step is performed on a computer. Based on preprocessing, effective spectral information is deeply mined, feature dimensions are expanded, and a multi-dimensional feature set is constructed. First, based on the preprocessed absolute spectral reflectance, vegetation indices related to crop SPAD value and nitrogen content are calculated. At the same time, the absolute spectral reflectance of the core bands sensitive to nutrient indicators is selected. Then, interactive features between sensitive bands and vegetation indices, difference features of red-edge bands, and coupling features of vegetation indices are constructed. Statistical features such as standard deviation, skewness, and coefficient of variation of the core bands are extracted. Through multi-dimensional feature fusion, the model's fitting and discriminative capabilities are enhanced.

[0097] S103: Abnormal Sample Removal and Feature Optimization: This step is performed on a computer, with the input being the full multidimensional feature set constructed in step S92. To avoid interference from outlier data and prevent feature redundancy and the curse of dimensionality, a quantile-adaptive outlier filtering strategy is first adopted. A threshold is adaptively set based on the prediction error to remove outlier samples that exceed the threshold, ensuring the regularity of the modeling data. Then, a three-layer progressive feature optimization is carried out: the first layer uses mutual information regression to remove low-correlation redundant features; the second layer uses recursive feature elimination combined with cross-validation to obtain effective features through iterative filtering; the third layer filters based on feature importance scores, ultimately obtaining a small but refined subset of core features, which, while ensuring feature discriminative power, suppresses model overfitting and improves computational efficiency.

[0098] S104: Model Building and Optimization This step is performed on a computer. Based on the core feature subset obtained in S103 and the true values ​​of SPAD and nitrogen content obtained using the SPAD-502 chlorophyll meter (a commercial chlorophyll measuring instrument) and the Kjeldahl method (a classic method for determining protein / nitrogen content), an XGBoost regression algorithm (Extreme Gradient Boosting) is used to construct a crop SPAD value and nitrogen content prediction model. The root mean square error is used as the core optimization index, and the model performance is comprehensively evaluated by combining the coefficient of determination, mean absolute error, and other indicators. Simultaneously, the traditional brute-force optimization method is abandoned, and a Bayesian optimization algorithm is used to refine hyperparameter optimization. A reasonable hyperparameter search range is set, and the cross-validation error is used as the optimization objective to intelligently search for the optimal parameter combination, accurately match the optimal model structure, and optimize the model training parameters to improve model robustness and prediction accuracy.

[0099] S105: Model Lightweighting and Device Deployment: To meet the demands of portable and real-time field monitoring, the prediction model was lightweighted and deployed on the device. First, the same data preprocessing logic as on the computer was deployed on the monitoring device, followed by the deployment of the prediction model. During deployment, the optimal trained model was lightweighted, exporting a lightweight binary file adapted for the ARM64 architecture Linux system. This compressed the model size and improved inference speed to meet real-time field monitoring requirements. Simultaneously, a feature name mapping dictionary was constructed to avoid embedded-side coding errors. A feature normalizer was deployed concurrently to ensure complete consistency between the embedded and PC data processing flows. Finally, the model was deployed to the main control board, achieving a closed-loop process from spectral data input to nutrient index output, enabling real-time non-destructive testing in the field.

[0100] Figure 5This image compares the segmentation results of different image fusion methods on soybean leaves, focusing on verifying the segmentation optimization effect of the fusion method proposed in this application on the subsequent target region. The original single-channel spectral image lacks color information and has low illumination feature recognition, making it difficult to distinguish effective leaf regions and prone to missed or incorrect segmentation. Traditional YCbCr fusion methods can restore image colors, but lack edge attention, spectral brightness normalization, and spectral quality perception, resulting in problems such as edge misalignment, ghosting, and missed segmentation, failing to meet the requirements of high-precision leaf segmentation. Compared to the original single-channel spectral image and the traditional YCbCr channel replacement fusion method, the multi-scale edge attention dual-weighted fusion image proposed in this application relies on a complete multi-module image fusion architecture. It retains the original color restoration advantage of the YCbCr color space conversion module, while adding an edge attention module to correct image registration deviations, optimizing image illumination distribution through a spectral brightness normalization module, and retaining effective image features through a spectral quality perception module. Finally, global image optimization is completed through a dual-weighted adaptive weighted fusion module. This multi-module fusion scheme can enhance leaf edge details, suppress shadow noise and background leaf interference, reduce missed segmentation and missegmentation errors, improve leaf segmentation accuracy and algorithm robustness, and provide reliable image data support for the subsequent accurate inversion of crop nutrient indicators.

[0101] Example 3 This embodiment provides a portable crop nutrient index detection device based on spectral images. Figures 6 to 9 The diagrams shown in the images are, in order: a first-view overall structural diagram, a first-view internal structural diagram, a second-view overall structural diagram, and a second-view internal structural diagram of the portable crop nutrient index detection device described in this embodiment. The components and hardware shown in the four diagrams are, in order: laser ranging module 1, visible light camera 2, spectral camera 3, Type-C charging port 4, communication and positioning module 5, slave control board 6, power switch 7, power control board 8, main control board 9, screen expansion board 10, Mobile Industry Processor Interface-Display Serial Interface (MIPI-DSI) display screen 11, 3000mAh lithium battery 12, power wake-up button 13, other top buttons 14, external antenna interface 15, mainboard buttons (including directional buttons 16, acquisition buttons 17, and confirmation buttons 18), and attitude sensor 19. To clearly illustrate the core technical solution of this application, the accompanying drawings only show the main structural and functional modules related to the inventive point; some conventional modules, auxiliary components, and known structures are not shown.

[0102] The device features a portable, handheld integrated structure with a unibody protective shell. Internally, it integrates six functionally partitioned mechanisms: main control, data acquisition, display, control, auxiliary sensing, and power supply. Dedicated holes are provided for each interface and button location on the shell, ensuring functionality while enhancing overall protection and durability for field operations. The relative spatial positions and layout of each hardware component on the shell are as follows: The frontmost surface of the housing is the spectral acquisition and auxiliary sensing area, where the laser ranging module 1, visible light camera 2, and spectral camera 3 are sequentially mounted side by side. The laser ranging module 1 is located on the left, the spectral camera 3 in the center, and the visible light camera 2 on the right. All three are on the same plane and perpendicular to the device, facing outwards (in the detection direction), ensuring simultaneous alignment with the target crop leaf during acquisition. A Type-C charging port 4 is embedded in the right side of the housing, flush with the housing surface, balancing protection and charging convenience. A grid-like heat dissipation window is located at the bottom of the front end to meet the heat dissipation requirements of the main control mechanism.

[0103] The interior of the housing houses the core control and power supply area, arranged in layers front-to-back and left-to-right. From a first-person perspective, the components mounted from front to back behind the camera and ranging module are: power control board 8, communication and positioning module 5, main control board 9, screen expansion board 10, and MIPI-DSI display screen 11. The power control board 8 is connected to the main control board 9 and the 3000mAh lithium battery 12, enabling power management and charge / discharge control; a Type-C charging port 4 is located on the right side for external charging. The 3000mAh lithium battery 12 is fixed at the bottom of the housing, directly below the main control board 9 and the slave control board 6, providing continuous power to the entire device. The communication and positioning module 5 is fixed behind the power control board 8 and in front of the main control board 9, near the front of the housing, corresponding to the external antenna interface 15, ensuring signal transmission stability. The main control board 9 is fixed behind the communication and positioning module 5 and is the core control unit of the device. The slave control board 6 is mounted to the left of the main control board 9 and connected to it, used to share the data acquisition and analysis tasks of peripherals, reducing the computational load on the main control board. The attitude sensor 19 is integrated on the slave control board 6 and is on the same plane as the main board buttons (direction buttons 16, acquisition buttons 17, and confirmation buttons 18), used to acquire device attitude data in real time. The screen expansion board 10 is located behind the main control board 9 and is attached to the back of the MIPI-DSI display screen 11, electrically connected to the display screen to realize interface display and touch interaction.

[0104] The rearmost end face of the housing is the display and control area. From a second-view perspective, the MIPI-DSI display screen 11 is embedded in the middle left area of ​​the rear of the housing, with its surface flush with the housing surface, balancing display clarity and protection. It is used to display the data acquisition and data analysis interfaces. The right side of the MIPI-DSI display screen 11 integrates motherboard buttons (directional buttons 16, acquisition buttons 17, and confirmation buttons 18), allowing for real-time control in conjunction with the screen display. A window is located on the upper left side of the housing for other control functions, corresponding from left to right to the power switch 7, power wake-up button 13, and other top buttons 14. The overall button layout conforms to handheld operation habits, facilitating one-handed operation of acquisition, switching, and calculation. The top window of the housing also allows direct debugging and expansion of the main control board via interfaces such as Universal Serial Bus (USB), Type-C, serial port, and High Definition Multimedia Interface (HDMI).

[0105] The connection relationships between the components are as follows Figure 10As shown. The spectral camera 3 and the communication positioning module 5 are connected to the main control board 9 via USB. The visible light camera 2 is connected to the main control board 9 via its dedicated Mobile Industry Processor Interface - Camera Serial Interface (MIPI-CSI interface). The energy storage unit (i.e., the 3000 mAh lithium battery 12) is connected to the power control board 8 via the VBAT pin for charging and discharging management. The power control board 8 is connected to the power wake-up button 13 via the KEY pin to enable power wake-up; simultaneously, the power control board 8 provides a stable 5V voltage to the main control board 9 and various peripherals via its power supply pin. Other buttons 14 on the top are brought out via the general purpose input / output (GPIO) pins of the main control board 9 to enable other extended functions. The screen expansion board 10 is connected to the main control board 9 via its dedicated MIPI-DSI interface and touch panel (TP) interface to enable bidirectional data and control signal transmission, and simultaneously transmits display data to the MIPI-DSI display screen 11 via the MIPI interface for display. The mainboard buttons (direction buttons 16, acquisition buttons 17, and confirmation buttons 18) are connected to the control board 6 via GPIO pins. Both the laser ranging module 1 and the attitude sensor 19 communicate with the control board 6 via an Inter-Integrated Circuit (I2C) communication method. Simultaneously, the control board 6 achieves bidirectional communication with the main control board 9 via a serial port: on the one hand, it receives control signals from the main control board 9; on the other hand, it sends back the received mainboard button signals, as well as the acquired and processed device distance and attitude data, to the main control board 9.

[0106] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting crop nutrient indicators based on spectral images, characterized in that, The method includes: In response to user-triggered start and acquisition commands, it acquires raw visible light images, raw spectral images, and working distance values; Based on the working distance value and the preset fixed registration parameters, the original visible light image is unidirectionally mapped to the original spectral image to generate a registered visible light image; An image fusion operation is performed on the original spectral image and the registered visible light image to generate a fused image. The image fusion operation is based on the acquired multi-scale edge attention weights and spectral quality weights. The original chroma channel of the normalized original spectral image and the original chroma channel of the registered visible light image are adaptively weighted and fused to generate a fused image with optimized luminance channel and restored to the red-green-blue color space through inverse color space transformation. The target leaf region of the fused image is segmented using a pre-defined improved lightweight semantic segmentation model to generate a binary mask matrix; Based on the binarized mask matrix, the original spectral digital quantization value corresponding to the effective pixel position is extracted from each spectral band of the acquired original spectral data. The first average original spectral digital quantization value of each spectral band is calculated, and the average relative spectral reflectance of each spectral band is calculated based on the first average original spectral digital quantization value of each spectral band. The first average original spectral digital quantization value is the average original spectral digital quantization value of the target crop area. Based on the average relative spectral reflectance of each spectral band, a preset lightweight nutrient index prediction model is used to invert nutrient indexes and output the predicted values ​​of crop nitrogen content and leaf relative chlorophyll content.

2. The method for detecting crop nutrient indicators based on spectral images according to claim 1, characterized in that, The fixed registration parameters include a fixed lateral offset and a fixed longitudinal offset of the target center between the visible light camera and the spectral camera, as well as a lateral pixel mapping scaling factor and a longitudinal pixel mapping scaling factor that are dynamically updated with the working distance. Based on the working distance value and the preset fixed registration parameters, the original visible light image is unidirectionally mapped to the original spectral image to generate a registered visible light image, specifically including: Based on the working distance value, the horizontal pixel mapping scaling factor, the vertical pixel mapping scaling factor, the horizontal fixed offset of the target center, and the vertical fixed offset of the target center, a third-order registration matrix with distance compensation is constructed. Using the third-order registration matrix, homogeneous coordinate transformation is performed on the coordinates of each pixel in the visible light image to generate the registered visible light image.

3. The method for detecting crop nutrient indicators based on spectral images according to claim 1, characterized in that, An image fusion operation is performed on the original spectral image and the registered visible light image to generate a fused image, specifically including: The original single-channel spectral image is expanded into a three-channel spectral image through channel replication; Using a preset multi-scale Laplacian convolution kernel, edge features are extracted from the three-channel spectral image and the registered visible light image respectively, to obtain spectral edge feature maps and visible light edge feature maps at each scale; The edge difference between the spectral edge feature map and the visible light edge feature map is calculated at each scale to obtain the edge difference map at each scale. The edge difference maps at all scales are then fused using multi-scale features to generate a fused edge difference map. Calculate pixel-wise normalized similarity based on the fused edge difference map to generate an edge similarity map. Perform nonlinear mapping and linear transformation processing on the edge similarity map to generate an edge attention weight map. The registered visible light image is converted from the original color space to a luminance and chrominance separated color space, separating the original luminance channel and the original chrominance channel; The pixel value distribution of the original spectral image is obtained, and the original spectral image is normalized by percentiles based on the statistical characteristics of the pixel value distribution to generate a normalized spectral brightness image. The original spectral image is subjected to local texture sharpness evaluation to generate a spectral quality weight map; the spectral quality weight map is used to characterize the imaging quality of the original spectral image at each pixel location; The edge attention weight map and the spectral quality weight map are linearly weighted and fused to obtain a weighted fusion result. The weighted fusion result is then subject to numerical range constraints to generate a comprehensive adaptive weight map. The normalized spectral brightness image and the comprehensive weight map are subjected to pixel-by-pixel weighted calculation to generate an optimized brightness channel; The optimized luminance channel replaces the original luminance channel of the registered visible light image, and combined with the retained original chroma channel, the image is restored to a fused image in red-green-blue color space through inverse color space transformation.

4. The method for detecting crop nutrient indicators based on spectral images according to claim 1, characterized in that, The improved lightweight semantic segmentation model is an improved YOLOv5-seg model. The improvements of the improved YOLOv5-seg model compared to the standard YOLOv5-seg model include: replacing the sigmoid weighted linear unit activation layer in the standard YOLOv5-seg model with a linear rectified activation layer, removing redundant post-processing modules in the standard YOLOv5-seg model, and pre-setting exclusive export parameters, so that the improved YOLOv5-seg model can directly output an open neural network exchange general format model that conforms to the quantization specifications of neural network inference engines.

5. The method for detecting crop nutrient indicators based on spectral images according to claim 1, characterized in that, Based on the binarized mask matrix, the original spectral digital quantization values ​​corresponding to the effective pixel positions are extracted from each spectral band of the acquired original spectral data. The first average original spectral digital quantization value for each spectral band is calculated, and the average relative spectral reflectance for each spectral band is calculated based on the first average original spectral digital quantization value. Specifically, this includes: For each spectral band of the acquired raw spectral data, a relative spectral reflectance acquisition operation is performed to obtain the average relative spectral reflectance of each spectral band; wherein, the relative spectral reflectance acquisition operation specifically involves: According to the binarized mask matrix, the original spectral digital quantization values ​​corresponding to the positions with a mask value of 1 are accumulated to obtain the original spectral digital quantization accumulated value. The positions with a mask value of 1 are taken as valid pixels, and the number of valid pixels is counted. Based on the original spectral digital quantization accumulated value and the number of valid pixels, the first average original spectral digital quantization value is calculated. Acquire whiteboard data; the whiteboard data includes the second average original spectral digital quantization value for each spectral band, and the second average original spectral digital quantization value is the average original spectral digital quantization value of the calibration whiteboard. Dark current correction is performed on the first average original spectral digital quantization value and the second average original spectral digital quantization value of the corresponding spectral band to obtain the corrected first average original spectral digital quantization value and the corrected second average original spectral digital quantization value. Based on the corrected first average original spectral digital quantization value and the corrected second average original spectral digital quantization value, the average relative spectral reflectance is calculated.

6. The method for detecting crop nutrient indicators based on spectral images according to claim 1, characterized in that, Based on the average relative spectral reflectance of each spectral band, a pre-set lightweight nutrient index prediction model is used to retrieve nutrient indexes, outputting predicted values ​​for crop nitrogen content and relative chlorophyll content in leaves, specifically including: Based on the average relative spectral reflectance of each spectral band and the actual physical reflectance of the calibration whiteboard obtained for the corresponding spectral band, the average absolute spectral reflectance of each spectral band is calculated. The average absolute spectral reflectance of each spectral band was smoothed using a Savitzky-Gore smoothing filter with a window length of 7, resulting in smoothed spectral data for each spectral band. Perform a first-order derivative transformation on the smoothed spectral data for each spectral band to obtain the transformed spectral data for each spectral band. The spectral data after transformation of each spectral band are subjected to standard normal transformation to obtain the spectral characteristics of each spectral band; The spectral features of each spectral band are input into the lightweight nutrient index prediction model, which outputs the predicted values ​​of crop nitrogen content and relative chlorophyll content in leaves. The lightweight nutrient index prediction model is a model constructed based on the XGBoost regression algorithm and obtained by Bayesian optimization of hyperparameters.

7. A portable crop nutrient index detection device based on spectral images, characterized in that, The device includes: The main control mechanism includes a main control board and a slave control board, the main control board being connected to the slave control board, and the main control board being used to execute the crop nutrient index detection method based on spectral images as described in any one of claims 1 to 6; A spectral acquisition mechanism includes a visible light camera and a spectral camera, which are respectively connected to the main control board. The visible light camera is used to acquire raw visible light images, and the spectral camera is used to acquire raw spectral images. An auxiliary sensing mechanism is provided, comprising a laser ranging module and a communication positioning module. The laser ranging module is connected to the slave control board, and the communication positioning module is directly connected to the main control board. The laser ranging module is used to collect working distance values ​​in real time and transmit the working distance values ​​to the main control board through the slave control board. The communication positioning module is used to acquire the positioning data of the device and realize data interaction communication. The display mechanism is connected to the main control board and is used to display the test results and the human-machine interface. The control mechanism includes physical buttons and a touch unit. The physical buttons are connected to the main control board and the slave control board respectively, and the touch unit is connected to the main control board. Both the physical buttons and the touch unit are used to receive user operation commands. The power supply mechanism is electrically connected to the main control board, the spectrum acquisition mechanism, the display mechanism, and the slave control board, and is used to provide working power.

8. The portable crop nutrient index detection device based on spectral images according to claim 7, characterized in that, The human-computer interaction interface includes a data acquisition interface and a data analysis interface. The data acquisition interface is used to display the original visible light image, the original spectral image, the working distance value, the positioning data, and the attitude data in real time. The data analysis interface is used to display the fused image, the binarized mask matrix, and the detection results.

9. The portable crop nutrient index detection device based on spectral images according to claim 7, characterized in that, The visible light camera is connected to the main control board via a mobile industry processor interface-camera serial interface; the spectral camera is connected to the main control board via a universal serial bus interface through an adapter; the display mechanism includes a mobile industry processor interface-display serial interface display screen, which supports multi-touch.

10. The portable crop nutrient index detection device based on spectral images according to claim 7, characterized in that, The auxiliary sensing mechanism also includes an attitude sensor, which is integrated on the slave control board. The attitude sensor is used to collect roll and pitch angles in real time and transmit them to the main control board via the slave control board.

Images