A light condition adaptive soil moisture estimation method based on multi-modal data fusion

Abstract

The application relates to the field of robot terrain perception, and proposes a light condition adaptive soil moisture estimation method based on multi-modal data fusion. First, an optimal exposure time calculation method is proposed to construct a multi-modal soil moisture dataset. Second, the SS-GAN algorithm is used to remove soil image shadows, and the channel attention mechanism ECA and the standard convolution layer are improved in combination with environmental information, ResNet-50 is embedded as the network backbone, and a soil moisture estimation submodel suitable for different light conditions is designed. Then, the improved loss function is used to train each submodel to adapt to specific light conditions. Finally, a submodel fusion and switching method is proposed to generate a soil moisture estimation map through gray value mapping. Through the improved neural network model for multi-modal data fusion and the design of the fusion and switching algorithm of the submodel, the soil moisture estimation precision of the robot under complex light conditions can be improved, and the terrain perception capability can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot terrain perception, and in particular to a soil moisture estimation method based on multi-modal data fusion under adaptive lighting conditions. BACKGROUND

[0002] In the field of robot terrain perception and autonomous navigation, accurate estimation of soil moisture is crucial for ensuring the safety and efficiency of robot travel on wet and slippery surfaces. Changes in soil moisture directly affect the interaction characteristics between the robot and the ground, including traction, grip, and friction, which are key factors affecting the safety and efficiency of robot travel. Especially in application scenarios such as agriculture, rescue, and exploration, robots need to adapt to changing soil conditions, and accurate soil moisture perception helps to optimize travel strategies and path planning.

[0003] Traditional soil moisture measurement methods, such as using insertable soil moisture sensors or drying methods, can provide relatively accurate humidity data in local areas, but have significant limitations in large-scale or complex terrain. These methods require direct contact with the soil, are complex and time-consuming to operate, and cannot achieve real-time monitoring, making it difficult to reflect the dynamic changes of the environment in a timely manner. In addition, the cost of equipment is high, which is not conducive to large-scale deployment on robot platforms. On the other hand, although satellite and aerial images have been applied to soil moisture monitoring in large-scale agriculture, their spatial and temporal resolution is low, and the update frequency is insufficient, making it difficult to meet the demand for accurate, real-time, and in-situ moisture measurement by robots.

[0004] In recent years, soil moisture estimation methods based on vision have gradually attracted attention. These methods mainly use single-modal data captured by RGB cameras perpendicular to the soil to estimate soil moisture through image processing and machine learning techniques. However, existing methods usually assume fixed lighting conditions and do not fully consider the various lighting changes that robots may encounter during actual travel, such as changes in lighting intensity and color temperature, as well as complex environmental factors such as shadows and reflections. In actual applications, the variability of lighting conditions can lead to a decrease in image quality, making feature extraction difficult, and thus significantly reducing the accuracy of humidity estimation. In addition, existing methods lack effective fusion of multi-modal data, such as infrared images, converted images in different color spaces, and environmental sensor data. This limits the robustness and applicability of the model in complex environments, and cannot meet the demand for accurate estimation of soil moisture by robots in variable environments.

[0005] In view of the existing challenges, it is necessary to develop a new soil moisture estimation method. This method will integrate multi-modal data collected under different lighting and viewing angle conditions, and use a condition-adaptive network to dynamically adapt to lighting changes, improving the accuracy and stability of soil moisture estimation. This enables robots to achieve safer and more efficient autonomous navigation in wet and slippery soil environments. SUMMARY

[0006] The application provides a light condition adaptive soil moisture estimation method based on multi-modal data fusion, which can accurately estimate soil moisture under various light conditions, and is particularly suitable for autonomous navigation and environmental monitoring of robots on soil roads. Specifically, the following steps are included:

[0007] Optimal exposure time calculation: a weighted fusion texture evaluation index is designed, which comprehensively considers the image GLCM energy (GLCM Energy), contrast, and information entropy. The index dynamically adjusts the weight coefficients according to the light intensity and soil moisture. On this basis, the calculation formula of the optimal exposure time is designed. Using a D435i camera to take soil images under different light conditions, set different exposure times. Calculate the GLCM Energy, contrast, and information entropy of the image under each exposure time, and determine the optimal exposure time through the calculation formula to ensure that the obtained image has the best texture features, thereby providing information-rich input data for subsequent soil moisture estimation.

[0008] Data acquisition and construction: a soil container placement scheme is designed to ensure that the layout covers the camera's field of view, so that the trained model can be applied to the entire field of view of the camera. According to the optimal exposure time determined under each light condition, soil images and related environmental data are collected to construct a multi-modal data set. Data augmentation and labeling are performed on the data set to enrich the training samples and improve the generalization ability of the model.

[0009] Network model design: an improved conditional channel attention (C-ECA) considering environmental information and a convolutional generative network (C-Conv) considering light conditions are designed, and these two improved networks are embedded in ResNet-50 as the network backbone. Based on the network backbone, according to the characteristics of soil images under different light conditions, specific network modules are embedded on the backbone network, and four soil moisture estimation sub-models are designed: back light sub-model, side light sub-model, and overcast sub-model. Each sub-model estimates soil moisture for images under specific light conditions.

[0010] Sub-model fusion / handover algorithm design: This algorithm includes a cloudy day recognition module and a sub-model weight distribution module based on fuzzy rules. First, a cloudy day recognition module implemented by a convolutional neural network (CNN) is introduced to determine whether it is a cloudy day to decide whether to call the cloudy day sub-model. In the weight distribution module, the angle θ between the driving direction and the direct sunlight direction, the light intensity I, and the side light sub-model weight α are used as fuzzy language variables. The membership function and fuzzy rules are designed considering the relationship between the driving direction and the adjacent sub-model. Through fuzzy reasoning and defuzzification, the weight coefficient α of the side light sub-model is obtained, and then the weight of the backlight sub-model or the backlight sub-model fused with the side light sub-model is determined.

[0011] Loss function improvement: The cross-entropy loss function is optimized by introducing a light weight to reflect the influence of different light conditions on soil moisture estimation. This light weight is dynamically adjusted according to the current light intensity, making the model more sensitive to errors under high light conditions, thus improving the estimation accuracy. In addition, to make the fully connected neural network in the conditional channel attention mechanism (C-ECA) and the conditional convolution (C-Conv) generate better channel weights and convolution kernels, a regularization term is introduced into the loss function. By imposing constraints on network parameters, the regularization term can prevent overfitting and improve the stability and generalization ability of the model under complex lighting conditions.

[0012] In practical applications, the trained network model is deployed on a wheeled robot to estimate soil moisture from real-time collected soil images. Based on the estimation results, a soil moisture estimation map is generated by mapping the gray value to visualize the spatial distribution of soil moisture. This method can improve the robot's perception of soil road environment, thereby enhancing its autonomous decision-making and navigation ability in complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 Light condition adaptive soil moisture estimation method schematic diagram;

[0014] Figure 2 Light condition adaptive soil moisture estimation method flowchart;

[0015] Figure 3 Soil humidity placement scheme diagram;

[0016] Figure 4 Simulated driving direction diagram;

[0017] Figure 5 SS-GAN shadow removal network model diagram;

[0018] Figure 6 Network backbone diagram;

[0019] Figure 7 Inverse photon model diagram;

[0020] Figure 8 Side photon model diagram;

[0021] Figure 9 Back photon model diagram;

[0022] Figure 10 Overcast day photon model diagram.

[0023] Figure 11 Fuzzy rule table diagram. DETAILED DESCRIPTION

[0024] Embodiments of the present application will now be described in further detail with reference to the accompanying drawings and examples.

[0025] An embodiment of the present application provides a soil moisture estimation method based on multi-modal data and a condition adaptive network, as shown in Figure 1 Figure 2 is a flowchart diagram according to the present application, and the method comprises the following steps:

[0026] ​The determination of the optimal exposure time is based on the variable lighting conditions that the robot may encounter in actual driving. First, the light intensity is divided into three levels from 10,000 lux to 100,000 lux, with the color temperature adjusted to the actual environment, setting three color temperature levels of 2,800 K, 4,500 K, and 5,500 K. Specifically, the color temperature is set to 2,800 K under 10,000 lux light intensity, suitable for natural light conditions in the morning or evening, simulating the soft light of sunrise or sunset; the color temperature is set to 4,500 K under 20,000 lux and 30,000 lux light intensity, suitable for the moderate light environment in the morning or afternoon; and the color temperature is uniformly set to 5,500 K under light intensity from 40,000 lux to 100,000 lux, to adapt to the high-intensity light of noon or midsummer. These settings ensure realistic simulation of lighting conditions at different times. In addition, to simulate the light changes under overcast conditions, the overcast light intensity is divided into three levels, 10,000 lux, 20,000 lux, and 30,000 lux, with the color temperature uniformly set to 6,500 K to reflect the higher color temperature and cold-toned light of overcast days. 10,000 lux simulates overcast morning or evening, 20,000 lux is suitable for overcast morning or afternoon, and 30,000 lux corresponds to overcast noon. These overcast lighting settings, in contrast to the high light intensity and lower color temperature of sunny conditions, allow the lighting effects of sunny and overcast days to be clearly distinguished. Through such light intensity and color temperature settings, various lighting conditions encountered by the robot in sunny and overcast environments can be effectively simulated, ensuring that the optimal exposure time selection can adapt to various natural lighting conditions. In addition, the position of the sun is divided into five azimuth angles, east, southeast, south, southwest, and west, which is closely related to the change of geographical position and time, providing a full range of light simulation for the environment perception system. Combined with five soil moisture levels (0%, 10%, 20%, 30%, 40%) and eight soil container placement modes, as shown in Figure 3 Eight potential driving directions (east, west, south, north, southeast, southwest, northeast, northwest), as shown in Figure 4 For each combination, a preliminary exposure time interval is first estimated as the basis for subsequent adjustment. Then, the automatic exposure time program is adjusted step by step, reducing by 1 microsecond (us) each time, and recording 20 photos. To ensure consistency of the data, only the texture indicators of the last 10 photos in each group setting are averaged. The texture richness of the images is evaluated by calculating the GLCM Energy, Contrast, and Information Entropy of these images, and the weighted fusion texture evaluation index formula is as follows:

[0027] V = w C (I, M) · C(T) + w E (I, M) · E(T) + w H (I, M) · H(T),

[0028] where V represents the texture richness, C(T), E(T), H(T) represent the contrast, GLCM Energy, and information entropy calculated from the soil image at exposure time T, respectively; w C , w E , w H are the weight coefficients corresponding to the contrast, GLCM Energy, and information entropy. Under high humidity conditions, the soil surface is smoother, and the change of GLCM Energy is more significant, so the weight should increase with the increase of humidity M. Under high light intensity, the details of the image are richer, and the change of information entropy H is more obvious, so the weight should increase with the increase of light intensity L. The contrast C as a basic feature, the weight can be set as a constant, or fine-tuned according to the requirements. The calculation formula is as follows:

[0029]

[0030] where k C , k E , k H are constant coefficients for adjusting the baseline influence of each feature, I threshold , M threshold are the light intensity and soil humidity thresholds. On this basis, the optimal exposure time is calculated as follows:

[0031]

[0032] where T opt is the calculated optimal exposure time.

[0033] The data collection scheme is as follows: first, mix water and soil to saturation, use a soil moisture sensor to measure the actual humidity, then divide the soil sample into 26 parts: 25 parts of soil are placed in the test site, the specific placement scheme is shown in Figure 3 , the main field of view of the camera is divided into 8 parts, then the soil container is placed in one of them, the soil container is composed of 25 rectangular soil containers, which is convenient to change the soil humidity. The other part of the soil is placed on the electronic scale to obtain the real-time field soil humidity. Then, set different light combinations (light intensity, color temperature, sun position, driving direction), soil humidity levels (0%, 10%, 20%, 30%, 40%), and car driving direction (east, west, south, north, southeast, southwest, northeast, northwest), as shown in Figure 4The optimal exposure time is preset under each lighting condition. Then, the RGB and IR images of the wet soil under the corresponding conditions are collected using the D435i camera, as well as the environmental variable configuration, to build a multi-modal dataset. The RGB images are converted into CIEXYZ and CIELCH color spaces using OpenCV to enhance the representation channels of the input images. The final modal dataset includes soil image data in various color spaces such as RGB, infrared (IR), CIEXYZ, CIELCH, as well as solar orientation data, light intensity, color temperature, robot driving direction, and camera exposure time. These soil image data from different color spaces and spectral ranges increase the representation dimension of soil features. In addition, the dataset combines changes in light intensity and color temperature, as well as solar orientation and robot driving direction, to simulate the imaging environment of the camera under different lighting conditions.

[0034] Data annotation and data augmentation. First, the collected images in the soil tank are annotated with accurate soil moisture values, and other areas in the image are marked as 0. Next, data augmentation methods are used to improve the robustness and generalization ability of the model: Gaussian noise is added to simulate minor noise caused by changes in lighting, salt and pepper noise is added to simulate sudden noise points that may be generated by sensors, light Gaussian blur is applied to enhance the model's feature extraction ability at different focal lengths, and motion blur is used to simulate image blur at different driving speeds to improve the model's adaptability to dynamic scenes. In addition, slight color jitter is performed in the CIEXYZ and CIELCH color spaces to enhance the model's robustness to color changes, especially to improve the detection accuracy of minor color differences caused by different soil moisture levels under the same lighting conditions. After data augmentation, the processed diversified dataset is used to train the soil moisture estimation sub-model, and the model performance is evaluated on the validation and test sets. Necessary optimizations are made based on the evaluation results to ensure the model's high stability and accuracy in practical applications. Throughout the process, the parameters of noise and blur need to be reasonably selected, and the amplitude of color jitter needs to be controlled to ensure that data augmentation effectively improves model performance without causing image information loss or introducing unnecessary bias, thereby achieving accurate detection and robust application of soil moisture.

[0035] The network model for soil shadow removal is based on the generative adversarial network (GAN) architecture, called soil shadow generative adversarial network (SS GAN). The SS GAN structure adopts a cyclic network form, including two generators (GXY and GYX) and two discriminators (DX and DY). The generator (Generator) and discriminator (Discriminator) are as follows: Figure 5The mutual mapping relationship between the two image domains is formed: the shadow image (X domain) and the non-shadow image (Y domain). The generator GXY is responsible for converting the shadow soil surface image into a non-shadow image, and the discriminator DY judges the true or false of the generated non-shadow image, and GYX performs the opposite operation to generate the corresponding shadow image from the non-shadow image. The model uses a cycle-consistent loss function and a content loss function to ensure the consistency of the generated image and the original image content, and an identity loss function is used to ensure that the generator will not change the content when inputting a non-shadow image. The network architecture is an unsupervised learning method, which does not need to train paired shadow and non-shadow images;

[0036] The network model is designed, and the specific implementation method is that the improved conditional channel attention C-ECA adds a fully connected layer based on the traditional ECA attention mechanism. The fully connected layer takes environmental information (light intensity I, light color temperature K, driving direction O, exposure time T, solar elevation angle H, and solar azimuth angle W) as input, and outputs a coefficient for adjusting the channel weight of the image. The coefficient is multiplied element by element with the channel weight output by the ECA mechanism to obtain the final channel weight, thereby realizing dynamic adaptation to environmental conditions and enhancing the model's ability to extract soil moisture features under different light conditions.

[0037] The convolution kernel generation network: through a fully connected convolution kernel generation network, the light condition information is encoded into a feature vector input, and the parameters of the convolution kernel are automatically adjusted through the fully connected neural network to realize dynamic generation of the convolution kernel. This method can optimize the convolution operation according to real-time environmental information, improve the feature extraction effect under each light condition, and enhance the robustness and generalization ability of the model.

[0038] Network backbone: characterized in that, the improved conditional convolution replaces a standard convolution layer in a residual block in ResNet-50, and the C-ECA is embedded after the four residual blocks, as Figure 6 shown.

[0039] Sub-model design: the backlight sub-model integrates the SKNet (Selective Kernel Networks) module, as Figure 7 shown. The SKNet dynamically selects convolution kernels of different sizes to adapt to multi-scale feature extraction, thereby capturing image details at different scales and further reducing the influence of strong light reflection on key features. The side light sub-model integrates the HiLo Attention (High-Low Attention) module, as Figure 8 shown. The backlight sub-model integrates the CFP (Centralized Feature Pyramid), as Figure 9The CFP can effectively capture global long-range dependencies through a centralized feature pyramid structure, and enhance feature diversity through an intra-layer feature adjustment mechanism. The HiLo Attention enhances the feature representation of high-contrast texture regions through the interaction of high-frequency and low-frequency features, ensuring that features related to soil moisture can be accurately extracted in areas with large changes in light. The Overcast Day Sub-model integrates an expansion convolution module for overcast conditions, as shown in FIG. 6. Due to the characteristics of low contrast, uniform illumination, and low light intensity of images in overcast environments, expansion convolution effectively handles weak texture and uneven brightness problems by expanding the receptive field, ensuring efficient extraction and enhancement of soil features, thereby improving the accuracy of soil moisture estimation. Figure 10

[0040] The model classification training process includes inputting the preprocessed multi-modal data (RGB images, CIEXYZ images, CIELCH images, IR images, light intensity, color temperature, solar azimuth angle, solar elevation angle, camera exposure time, driving direction) into the soil moisture estimation network. During the training process, in order to better adapt to changes in lighting conditions, a weighting factor related to light intensity is introduced based on the standard cross-entropy loss function. The weight factor is defined as:

[0041]

[0042] where α, β, γ, δ are adjustment factors, I is the current light intensity, I_max is the maximum value of light intensity, I threshold is the light intensity threshold. The loss function after introducing the light weight is as follows:

[0043] L C = -∑yC(I)log(p),

[0044] where the weighting factor C(I) ensures that the model is more sensitive to errors under special lighting conditions. On this basis, in order to make the fully connected neural network in C-ECA better generate channel weights, we can add a special regularization term to the loss function to encourage the sparsity or specific distribution of channel weights w C-ECA . The specific formula is as follows:

[0045] L C-ECA = λ C-ECA ·Regularization(w C-ECA ),

[0046] where Regularization(w C-ECA ) = ||w C-ECA ​||1 is a sparsity constraint, using L1 regularization to highlight important channel weights and make non-important weights approach zero. In order to make the fully connected neural network in C-Conv generate more suitable convolution kernels, a regularization term of the convolution kernel parameter K C-Conv is added to the loss function, and the specific formula is as follows:

[0047] L C-Conv =λ C-Conv ·Regularization(K C-Conv ),

[0048] wherein is a smoothness constraint, using L2 regularization to prevent the convolution kernel parameter from being too large and promote the stability of the model. Combining the above loss terms, the final loss function is as follows:

[0049] L total =L C +L C-ECA +L C-Conv ,

[0050] When dividing the data set, according to the angle between the driving direction of the robot and the direct sunlight, the training data is selected: when the angle between the driving direction and the direct sunlight is between 0° and 45°, this part of data is used to train the backlight sub-model; when the angle is between 45° and 135°, the data is used to train the side light sub-model; when the angle is between 135° and 180°, the data is used to train the backlight sub-model; for overcast conditions (low light intensity and high color temperature), the data is used to train the overcast sub-model. The optimizer can be selected as Adam or SGD, and the initial learning rate is set to 0.001, and a learning rate decay strategy is combined to adapt to the training process at different stages. In order to improve the generalization ability of the model, the cross-validation method is adopted, and the data set is divided into multiple parts, and one part is used as the validation set, and the rest is used for training. This method can effectively reduce the risk of overfitting and improve the adaptability and robustness of the model in practical application;

[0051] Fusion / handover algorithm, overcast recognition module, using ResNet network to recognize whether the weather is overcast, if detected as overcast condition, switch to overcast sub-model. Weight distribution module, based on the angle between the direction of the car and the direction of the sun (azimuth angle) and the dynamic change of light intensity, a set of fuzzy logic rules is designed, dynamically select or weight fusion three sub-models (backlight sub-model, side light sub-model, backlight sub-model) to estimate soil moisture. The overcast sub-model is switched by other methods, and this algorithm does not contain the calculation of the overcast sub-model. In any case, the angle θ must be between the angle range of the two sub-models, and the degree of angle deviation and light intensity together determine the weight of the two sub-models. The membership function of the angle is designed based on the degree of angle deviation: where the backlight angle membership function is as follows:

[0052]

[0053] The side light angle (left) membership function is as follows:

[0054]

[0055] The side light angle (right) membership function is as follows:

[0056]

[0057] The backlight angle membership function is as follows:

[0058]

[0059] The fuzzification of light intensity I is divided into three fuzzy sets, and the membership function is as follows:

[0060]

[0061] The membership function of the output weight α is as follows:

[0062]

[0063] The fuzzy rules are shown in Figure 11 ; based on the minimum-maximum fuzzy reasoning, and the centroid method is used to solve the fuzzy to obtain the final weight coefficient α model1 . And on this basis, the weighted fusion calculation is carried out to obtain the final estimation value, the formula is as follows:

[0064] Y final = α model1 · Y model1 + α model2 · Y model2 ,

[0065] where α model1 is the weight of the side light sub-model, αmodel2 = 1 - a model1 is another weight involved in the fusion model, Y model1 and Y model2 is the output result of the corresponding model;

[0066] In the inference stage, the system deploys the trained soil moisture estimation model to a mobile robot equipped with an Intel D435i camera, achieving real-time estimation of soil moisture. First, the robot collects RGB and infrared (IR) images of the soil surface at a frame rate of 30 FPS and a resolution of 640x480 through the Intel D435i camera, while monitoring the light intensity and color temperature in real time through the ambient light sensor, and adjusting the camera's exposure time T according to the lighting conditions to ensure image quality. In addition, the robot records the driving heading through the electronic compass module every second, with a heading error controlled within ±2°, and all data is processed on a NVIDIA Jetson AGX Xavier or equivalent edge computing device. The collected RGB images are converted to CIE XYZ and CIE LCH images, which, together with the standardized infrared images, form a multi-modal input data matrix. To eliminate the interference of ambient light and shadows, the system applies the SS-GAN (Shadow-Removal Generative Adversarial Network) algorithm, which removes shadows in the image through the generator network and ensures image quality through the discriminator. The SS-GAN hyperparameters are set to a learning rate of 0.0002, using the Adam optimizer, and the training duration is 100 epochs. Subsequently, the system applies a fusion switching algorithm to dynamically adjust the sub-model weights according to the real-time environmental lighting conditions. Finally, the different levels of soil moisture are mapped to grayscale values using the grayscale value, with soil moisture value S from 0% to 40% mapped to grayscale value G from 0 to 255, with a linear mapping relationship, and the calculation formula is as follows:

[0067]

[0068] where, when the soil moisture S is 0%, the grayscale value G is 0 (completely black); when the soil moisture S is 10%, the grayscale value G is 63.75 (dark gray); when the soil moisture S is 20%, the grayscale value G is 127.5 (medium gray); when the soil moisture S is 30%, the grayscale value G is 191.25 (light gray); and when the soil moisture S is 40%, the grayscale value G is 255 (completely white). Through this linear mapping method, different levels of soil moisture can be displayed intuitively.

Claims

1. A method for adaptive soil moisture estimation based on illumination conditions using multimodal data fusion, characterized in that, Includes the following steps: Step 1: Design a weighted fusion soil image texture evaluation index, which comprehensively considers the energy of the gray-level co-occurrence matrix (GLCM), contrast, and information entropy, and dynamically adjusts the weight coefficients according to the light intensity and soil moisture. Based on this evaluation index, formulate a calculation formula to determine the optimal exposure time of the camera under different lighting conditions. Step 2: A method for constructing a multimodal soil moisture dataset is proposed. Under different lighting conditions, the camera parameters are adjusted based on the optimal exposure time determined in Step 1. Combined with the robot's travel path and the actual lighting environment, a representative combination of lighting conditions and travel direction is designed. Soil containers are placed in batches to ensure coverage of the entire camera field of view. Soil images and related environmental data at different moisture levels are collected simultaneously. Multiple data modalities are integrated to construct a multimodal dataset. Subsequently, data augmentation and annotation are performed on the dataset. Step 3: Improve the ECA attention mechanism and standard convolutional layer by combining environmental information, design conditional channel attention C-ECA and conditional convolution C-Conv, and embed ResNet-50 as the network backbone. For the characteristics of soil images under different lighting conditions, design four sub-models: backlight sub-model, backlight sub-model, sidelight sub-model and cloudy sub-model. Multimodal data fusion is achieved through sub-models to estimate soil moisture under various lighting conditions. Step 4: Design an improved cross-entropy loss function. The cross-entropy loss function is optimized by introducing illumination weights, which are dynamically adjusted according to the current illumination intensity. In addition, in order to enable the fully connected neural networks in the Conditional Channel Attention (C-ECA) and Conditional Convolution (C-Conv) mechanisms to generate better channel weights and convolution kernels, a regularization term is introduced into the loss function. Step 5: Design a sub-model fusion / switching algorithm. This algorithm includes a cloudy day recognition module and a sub-model weight allocation module based on fuzzy rules. First, a cloudy day recognition module implemented by a convolutional neural network (CNN) is introduced to determine whether it is a cloudy day and whether the cloudy day sub-model needs to be called. In the weight allocation module, the angle θ between the driving direction and the direct sunlight direction, the light intensity I, and the side-photon sub-model weight α are used as fuzzy linguistic variables. Considering the relationship between the driving direction and adjacent sub-models, a membership function and fuzzy rules are designed. Through fuzzy inference and defuzzification, the weight coefficient α of the side-photon sub-model is obtained, and then the weight 1-α of the backlight sub-model or the reverse light sub-model fused with the side-photon model is determined. Step 6: Deploy the trained network model onto the wheeled robot, estimate the soil moisture content of the real-time collected soil images, and generate a soil moisture estimation map based on the estimation results through grayscale mapping.

2. The method for adaptive soil moisture estimation based on multimodal data fusion under illumination conditions according to claim 1, characterized in that, The formula for the weighted fusion texture evaluation index mentioned in step 1 is as follows: V=w C (I,M)·C(T)+w E (I,M)·E(T)+w H (I,M)·H(T), Where V represents texture richness, and C(T), E(T), and H(T) represent the contrast, GLCM energy, and information entropy calculated from the soil image at exposure time T, respectively; w C w E w H , where represents the weighting coefficients for contrast, GLCM Energy, and information entropy, and their calculation formulas are as follows: Where k C ,k E ,k H These are constant coefficients used to adjust the baseline influence of each feature. threshold M threshold Based on the threshold values ​​of light intensity and soil moisture, the optimal exposure time is calculated as follows: Where T opt The calculated optimal exposure time.

3. The method for adaptive soil moisture estimation based on multimodal data fusion under illumination conditions according to claim 1, characterized in that, The multimodal dataset mentioned in step 2 contains multiple modalities and their corresponding data. Specifically, the dataset includes: an image modality, which is soil image data acquired in the color spaces and spectral ranges of RGB, IR, CIEXYZ, and CIELCH, enriching the representation dimensions of soil features; an illumination modality, which includes data on light intensity and color temperature, recording changes in ambient lighting conditions; a location information modality, including solar azimuth data and robot heading, simulating the imaging environment of the camera in different times and spaces; and a camera parameter modality, which includes camera exposure time, recording the camera settings parameters during the imaging process.

4. The method for adaptive soil moisture estimation based on multimodal data fusion under illumination conditions according to claim 1, characterized in that, Step 3 describes C-ECA, which adds a fully connected layer to the traditional ECA attention mechanism. This fully connected layer takes environmental information, namely light intensity I, light color temperature K, driving direction O, exposure time T, solar altitude angle H, and solar azimuth angle W, as input to generate coefficients for adjusting the image channel weights. These coefficients are multiplied element-wise with the channel weights output by the ECA mechanism to obtain the final channel weights.

5. The method for adaptive soil moisture estimation based on multimodal data fusion under illumination conditions according to claim 1, characterized in that, The conditional convolution described in step 3 uses a fully connected convolutional kernel generation network to take environmental information, namely light intensity I, light color temperature K, driving direction O, exposure time T, solar altitude angle H, and solar azimuth angle W, as input to dynamically adjust the parameters of the convolutional kernel. Specifically, the light condition information is first processed by a multilayer perceptron to generate corresponding convolutional kernel weights, and the convolutional kernel parameters are automatically adjusted according to the current environmental conditions.

6. The method for adaptive soil moisture estimation based on multimodal data fusion under illumination conditions according to claim 1, characterized in that, The network backbone described in step 3 replaces the standard 3×3 convolutional layers in the residual blocks of ResNet-50 with improved conditional convolutions, and C-ECA is embedded after each residual block.

7. The method for adaptive soil moisture estimation based on multimodal data fusion under illumination conditions according to claim 1, characterized in that, In step 3, based on the characteristics of soil images under different lighting conditions, targeted network modules are embedded in each sub-model to improve the efficiency and accuracy of feature extraction: The backlighting model incorporates a selective convolutional kernel network module, which enhances the ability to capture multi-scale features under backlighting conditions by adaptively selecting convolutional kernels of different scales. Backlight sub-model: Embedded with a centralized feature pyramid module, it enhances the ability to extract detailed features under backlight conditions and improves sensitivity to low-contrast areas through the fusion and transfer of multi-level features. Side-light model: Embedded with high and low frequency attention modules, it simultaneously focuses on high-frequency and low-frequency information, enhances feature expression and discrimination capabilities under side lighting conditions, and effectively addresses image feature differences caused by uneven illumination. Cloudy Day Model: By embedding a dilated convolution module, it expands the receptive field and obtains more contextual information without significantly increasing the parameters, thereby enhancing the global feature extraction effect under cloudy and low-light conditions and improving the overall perception capability of the image.

8. The method for adaptive soil moisture estimation based on multimodal data fusion under illumination conditions according to claim 1, characterized in that, The improved cross-entropy loss function described in step 4 introduces a weighting factor related to light intensity based on the standard cross-entropy loss function. The weighting factor is defined as follows: Where α, β, γ, and δ are adjustment factors, and I is the current light intensity. max I represents the maximum light intensity. threshold The loss function after introducing illumination weights, with illumination intensity as the threshold, is as follows: L C =-∑yC(I)log(p), The weighting factor C(I) ensures that the model is more sensitive to errors under special lighting conditions. Based on this, to enable the fully connected neural network in C-ECA to generate channel weights better, a dedicated regularization term is added to the loss function to encourage channel weights w to... C-ECA The sparsity or specific distribution of [the property] is given by the following formula: L C-ECA =λ C-ECA ·Regularization(w C-ECA ), Where Regularization(w C-ECA )=||w C-ECA ||1 represents a sparsity constraint. L1 regularization is used to highlight the weights of important channels and make the weights of unimportant channels approach zero. To enable the fully connected neural network in C-Conv to generate more suitable convolutional kernels, the kernel parameter K is added to the loss function. C-Conv The regularization term, specifically the formula is as follows: L C-Conv =λ C-Conv ·Regularization(K C-Conv ), in To ensure smoothness, L2 regularization is used to prevent excessively large convolution kernel parameters and improve model stability. Combining the above loss terms, the final loss function is as follows: L total =L C +L C-ECA +L C-Conv 。 9. The method for adaptive soil moisture estimation based on multimodal data fusion under illumination conditions according to claim 1, characterized in that, The fuzzy linguistic variables mentioned in step 5 include the angle θ, which represents the angle between the car's driving direction and the direction of direct sunlight, ranging from [0, 180°], and divided into backlight angle, sidelight angle-left, sidelight angle-right, and backlight angle; the light intensity I ranges from [0, 100000 lux], and is divided into three levels: low light, medium light, and high light; and the output is the weight α of the side photon model, ranging from [0, 1], and divided into four levels: low, medium, high, and very high.

Images