Agricultural operation decision-making method and device based on multi-modal fusion

Abstract

The invention provides an agricultural operation decision-making method and device based on multi-modal fusion, and the method comprises the steps: carrying out the alignment processing of obtained multi-modal agricultural data, carrying out the adaptive Kriging space interpolation processing of soil conductivity data in the aligned multi-modal agricultural data, obtaining a farmland state data stream, and carrying out the prediction of the farmland state data stream; determining the multi-modal characteristic data; and determining the weight of each feature data in the multi-modal feature data according to the growth stage index of the target crop to obtain a multi-modal weight so as to determine a multi-modal feature vector, and inputting the multi-modal feature vector, the historical operation data of the target crop and a future environment prediction result into an operation decision model to obtain an operation decision scheme of the target crop. The problem that when automatic operation decision making is achieved through multi-source data fusion, due to the fact that data representation is difficult to achieve accurately due to the data fragmentation problem of multi-source data, the multi-source fusion effect is poor, and the accuracy of operation decision making is affected is solved.

Description

Technical Field

[0001] This invention relates to the field of agricultural data processing technology, and in particular to an agricultural operation decision-making method and apparatus based on multimodal fusion. Background Technology

[0002] Traditional agricultural operations, such as open-field vegetable production, have long relied on manual experience, resulting in low efficiency. With the development of smart agriculture technology, unmanned operation scenarios are gradually increasing, and unmanned equipment, such as unmanned seeders, intelligent spraying vehicles, and unmanned harvesters, are being introduced. By fusing multi-modal data, including crop and environmental data, and then making dynamic decisions based on the characteristics of this multi-source data, corresponding operational strategies are generated to control the equipment and achieve unmanned automated operations.

[0003] However, existing technologies still have significant shortcomings in multi-source data fusion and dynamic decision-making. On the one hand, the data acquisition systems used in existing technologies often rely on single sensors or single-dimensional data, such as temperature and humidity sensors or visible light cameras, making it difficult to comprehensively characterize crop growth status and dynamic changes in the crop growth environment. On the other hand, multi-source data suffers from data fragmentation, and traditional data fusion methods, such as weighted averaging or linear regression, struggle to handle the complex correlations between multimodal data. For example, the spatiotemporal resolution of hyperspectral images in crop growth data and agricultural machinery operation data is inconsistent; direct fusion can lead to information loss, resulting in poor multi-source data fusion effects and severely impacting the accuracy of operational decisions. Summary of the Invention

[0004] This invention provides an agricultural operation decision-making method and apparatus based on multimodal fusion, which solves the problem that existing technologies, when achieving automated operation decision-making through multi-source data fusion, suffer from poor multi-source fusion results due to the fragmentation of multi-source data, making it difficult to accurately represent the data and affecting the accuracy of operation decisions.

[0005] This invention provides a job decision-making method based on multimodal fusion, comprising the following steps: Acquire multimodal agricultural data to be processed, including crop growth data, growth environment data, and agricultural machinery operation data; The multimodal agricultural data is aligned, and adaptive Kriging spatial interpolation is performed on the soil electrical conductivity data included in the aligned multimodal agricultural data to obtain a farmland state data stream. The weights of each feature data in the multimodal feature data are determined based on the growth stage index of the target crop to obtain the multimodal weights. The multimodal feature vectors are then determined based on the multimodal weights. The multimodal feature data is extracted from the farmland state data stream. The multimodal feature vector, the historical operation data of the target crop, and the future environmental prediction results are input into the operation decision model to obtain the operation decision scheme for the target crop.

[0006] In some embodiments, the multimodal agricultural data is aligned in the following manner: Using the second pulse signal of the agricultural machinery operation data as a reference, the timestamps of the crop growth data and the growth environment data are clocked for synchronization. Based on the time of data collection of the crop growth data, the growth environment data and the agricultural machinery operation data are interpolated simultaneously. Align the crop growth data, the growth environment data, and the agricultural machinery operation data in a spatial coordinate system.

[0007] In some embodiments, adaptive kriging spatial interpolation is performed on soil electrical conductivity data included in the aligned multimodal agricultural data to obtain a farmland state data stream, including: An adaptive kriging spatial interpolation model is constructed based on the root density function of the target crop. The adaptive kriging spatial interpolation model is then used to perform adaptive kriging spatial interpolation on the soil electrical conductivity data included in the growth environment data to obtain the corresponding soil interpolation data. Spatial resolution is unified for multimodal agricultural data including the soil interpolation data; Determine the correlation gradient between soil parameters and environmental variables in the growth environment data, and construct a multimodal time-series compensation function based on the correlation gradient; The multimodal temporal compensation function is invoked to perform high-frequency prediction on low-frequency values ​​in the growth environment data after spatial resolution unification, thereby obtaining high-frequency predicted environment data; A farmland status data stream is constructed based on the soil interpolation data, the high-frequency predicted environmental data, the crop growth data after spatial resolution unification, and the agricultural machinery operation data.

[0008] In some embodiments, the adaptive kriging space interpolation model is represented by the following formula: in, This represents the root density function of the target crop. Indicates the target interpolation point. This indicates the known sampling points for soil electrical conductivity data, which are included in the growth environment data. This represents the soil interpolation data obtained after interpolating the target interpolation point. The weights represent the weights of the Kriging space interpolation algorithm, and n represents the number of target interpolation points. This represents the soil electrical conductivity data corresponding to the known sampling points. This represents the j-th target interpolation point.

[0009] In some embodiments, the feature data in the multimodal feature data includes: environmental features, crop physiological features, and agricultural machinery features. The step of determining the weights of each feature data in the multimodal feature data based on the growth stage index of the target crop to obtain the multimodal weights includes: Based on the environmental and physiological data corresponding to the target crop, the growth stage index of the target crop is determined; The environmental weights of the environmental features, the physiological weights of the crop physiological features, and the operational weights of the agricultural machinery features are dynamically allocated based on the growth stage index to obtain multimodal weights.

[0010] In some embodiments, determining the multimodal feature vector based on the multimodal weights includes: Each feature data in the multimodal feature data is input into a multilayer perceptron to obtain the output features; The output features are weighted and fused according to the multimodal weights to obtain a multimodal feature vector.

[0011] In some embodiments, before inputting the multimodal feature vector, the historical operation data of the target crop, and the future environmental prediction results into the operation decision model, the method further includes: The historical agricultural data relationships and historical operation records are determined as the historical operation data of the target crop. The historical agricultural data relationships include the relationship between spraying amount and crop yield, and the historical operation records include agricultural machinery failure records. The future meteorological information of the current growth environment of the target crop is determined as the future environmental prediction result. The future meteorological information includes the future rainfall probability and future wind speed change information of the current growth environment.

[0012] In some embodiments, the step of inputting the multimodal feature vector, the historical operation data of the target crop, and the future environmental prediction results into the operation decision model to obtain the operation decision scheme for the target crop includes: A job decision model is constructed based on short-term and long-term nodes of a dynamic Bayesian network. The multimodal feature vector, historical operation data of the target crop, and future environmental prediction results are input as particles into the operation decision model. The particle swarm optimization algorithm is called to update the particles to update the short-term nodes and the long-term nodes, thereby obtaining the operation decision scheme for the target crop. The weight of the particles is updated based on the time decay factor.

[0013] This invention provides a job decision-making device based on multimodal fusion, comprising the following modules: The acquisition module is used to acquire multimodal agricultural data to be processed, including crop growth data, growth environment data, and agricultural machinery operation data. The alignment module is used to align the multimodal agricultural data and perform adaptive Kriging space interpolation on the soil electrical conductivity data included in the aligned multimodal agricultural data to obtain a farmland state data stream. The determination module is used to determine the weight of each feature data in the multimodal feature data according to the growth stage index of the target crop, to obtain the multimodal weight, and to determine the multimodal feature vector based on the multimodal weight. The multimodal feature data is extracted from the farmland state data stream. The decision module is used to input the multimodal feature vector, the historical operation data of the target crop, and the future environmental prediction results into the operation decision model to obtain the operation decision scheme of the target crop.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the job decision method based on multimodal fusion as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the job decision method based on multimodal fusion as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the job decision method based on multimodal fusion as described above.

[0017] The present invention provides an agricultural operation decision-making method and apparatus based on multimodal fusion. This method unifies crop growth data, growth environment data, and agricultural machinery operation data through unified alignment processing. It also employs adaptive Kriging space interpolation processing for the included soil conductivity data, facilitating the unification of multimodal agricultural data into a single spatiotemporal coordinate system and enabling multi-dimensional data fusion to address the problem of data fragmentation. Furthermore, by determining the feature weights of the multi-source data using the growth stage index of the target crop, the fusion processing of various feature data is achieved, enabling an accurate representation of the current farmland state. Finally, based on the multimodal features, and combined with historical operation data and future environmental conditions, operation decisions are made. This ensures that the operation decision scheme adapts to the growth stage of the target crop and the future environment, improving the effectiveness and accuracy of agricultural machinery operation. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the job decision-making method based on multimodal fusion provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the operation decision-making device based on multimodal fusion provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] The following description, in conjunction with the accompanying drawings, describes the agricultural operation decision-making method and apparatus based on multimodal fusion of the present invention. Figure 1 This is a flowchart illustrating the job decision-making method based on multimodal fusion provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 101 to 104, which are described in detail below.

[0024] Step 101: Obtain the multimodal agricultural data to be processed.

[0025] Specifically, multimodal agricultural data mainly includes crop growth data, growth environment data, and agricultural machinery operation data. Growth environment data specifically refers to soil data, including soil temperature, soil moisture, electrical conductivity, and soil pH. This soil data is standard time-series data, which can be acquired through sensors mounted on agricultural machinery, with a recording frequency of once every 15 minutes. Crop growth data mainly includes crop growth stage, growth rate, and leaf area index (LAI), which can be determined by acquiring hyperspectral images of the target crop using hyperspectral cameras mounted on drones or satellites. The acquisition frequency is based on the stages of the crop's growth cycle. Agricultural machinery operation data mainly includes agricultural machinery location, operating status (speed, fuel consumption, and operating area), sensor data, and operating time. This data can also be acquired through sensors mounted on agricultural machinery, with a recording frequency of once every 15 minutes.

[0026] After multimodal agricultural data collection, preprocessing is generally required. First, timestamp standardization is performed: using Global Positioning System (GPS) time as the benchmark, the timestamps of all multimodal agricultural data are unified, the sampling frequencies of different devices are aligned, time granularity is matched through interpolation or downsampling, and missing timestamps are filled using linear interpolation or historical data. Second, spatial coordinate transformation is necessary to unify the data to the Universal Transverse Mercator (UTM) projection coordinate system, eliminating geographical coordinate differences. Multimodal agricultural data also needs format standardization, converting data in different formats (such as comma-separated values ​​(CSV), JavaScript Object Notation (JSON), and binary) into structured tables (e.g., time-space-attribute triples).

[0027] Furthermore, certain data in multimodal agricultural data require denoising and normalization. Specifically, Gaussian filtering and wavelet denoising are applied to hyperspectral images to eliminate image noise. Soil data are standardized using Z-Score (standard score) to eliminate dimensional differences, as shown in the following formula: (1) in, The mean value for the band; Standard deviation; For the original soil data, This is the standardized soil data.

[0028] Step 102: Align the multimodal agricultural data and perform adaptive kriging spatial interpolation on the soil electrical conductivity data included in the aligned multimodal agricultural data to obtain the farmland state data stream.

[0029] The alignment process here mainly includes three aspects: timestamp clock synchronization, interpolation alignment, and spatial coordinate system alignment, which will be explained in detail below.

[0030] The clock synchronization of timestamps specifically uses the second pulse signal of agricultural machinery operation data as a reference to synchronize the timestamps of crop growth data and growth environment data.

[0031] Here, the Global Navigation Satellite System (GNSS) is used for time synchronization to achieve a unified clock source for the data. The BeiDou Real-Time Kinematic (RTK) module, mounted on agricultural machinery, serves as a global spatiotemporal reference, providing a Pulse Per Second (PPS) signal as the unified clock source for all sensors, thereby eliminating clock drift errors. For example, the hyperspectral camera collecting crop growth data and the soil sensor collecting environmental data both use the GNSS PPS signal as a reference for timestamping. Furthermore, edge computing nodes receive GNSS time signals in real time and correct the timestamps of sensor data to ensure global synchronization of the data stream.

[0032] Interpolation alignment specifically uses the time of crop growth data collection as a benchmark to simultaneously interpolate growth environment data and agricultural machinery operation data.

[0033] Here, using the hyperspectral image acquisition time as a reference, linear or spherical interpolation is performed on agricultural machinery trajectory data and soil data to obtain equivalent values ​​at the same time. Furthermore, for agricultural machinery trajectory data within the agricultural machinery operation data, the precise coordinates of the hyperspectral image acquisition time can be calculated through linear interpolation based on the GNSS position information of two consecutive frames, achieving interpolation completion. For soil data, a sliding window method can be used to interpolate discrete sampling points, generating a continuous time series.

[0034] Spatial coordinate system alignment involves aligning crop growth data, growth environment data, and agricultural machinery operation data using spatial coordinate systems.

[0035] Specifically, the first step is coordinate system transformation. When acquiring hyperspectral images using a hyperspectral camera mounted on a drone or satellite, GNSS position and orientation system (POS) data is recorded simultaneously. Radiometric and geometric correction models (e.g., Remote Procedure Call (RPC) models) are then used to map the pixel coordinates of the hyperspectral images to WGS84 or UTM coordinate systems. The second step is spatial alignment. For the centimeter-level positioning data output in real-time by the RTK module mounted on agricultural machinery, this data can be directly converted to the target coordinate system to maintain consistency with the hyperspectral imagery and soil data.

[0036] In this embodiment of the invention, the alignment of multimodal agricultural data is achieved by performing clock synchronization of timestamps, interpolation alignment, and spatial coordinate system alignment, thereby ensuring the consistency of multimodal agricultural data in time and space and the synchronization of data streams.

[0037] In this embodiment of the invention, the hyperspectral imagery has a spatial resolution of 10cm, a wavelength range of 400-2500nm, and an acquisition frequency of 1 frame / second. Soil data related to the growth environment, including soil conductivity and organic matter content, is obtained through discrete point sampling at a sampling frequency of 1Hz, covering a soil layer from 0 to 50cm. Agricultural machinery trajectory data is obtained through BeiDou RTK positioning, with a positioning accuracy between -1cm and 1cm, and a sampling frequency of 100Hz. Environmental parameters related to the growth environment, such as temperature, humidity, and light intensity, are sampled at a frequency of 5Hz.

[0038] The adaptive Kriging spatial interpolation here is implemented using an improved Kriging interpolation optimization algorithm, which aims to interpolate the soil electrical conductivity data included in the multimodal agricultural data, thereby unifying the spatiotemporal resolution of the multimodal agricultural data.

[0039] Traditional Kriging interpolation has shortcomings and is unsuitable for agricultural scenarios. This is because the core assumption of traditional Kriging interpolation algorithms is that spatial attributes change stationarily and their correlation depends only on geometric distance. The closer the distance, the more similar the attributes of two points. However, the interpolation weights... The allocation of data is entirely determined by a semi-variogram model in mathematics. This model only cares about the straight-line distance between the sampling point and the target point, which is a kind of "blind" geometric interpolation and cannot perceive the behavior and effects of objects in space. In actual agricultural scenarios, target crops do not absorb water and fertilizer through "straight-line distance," but through their root network. Roots are not uniformly distributed in the soil, but have specific spatial configurations (such as taproot, lateral roots, and root hair zones). What is truly meaningful for crop growth is the soil environment within the effective absorption range of its roots. A point that is far from the root system but geometrically close may have a much smaller actual impact than a point that is slightly farther away but located within the root zone.

[0040] For example, target interpolation point Located in the center of the root system of a cabbage plant. A known sampling point is located 10 cm to its east (within the root system). Electrical conductivity is There is a sampling point 10 centimeters to its west (outside the root system). Electrical conductivity is When calculating using traditional Kriging interpolation, because... and and If their geometric distances are equal, they may obtain similar interpolation weights (e.g., ...). =0.45, =0.45), the final interpolation result is ≈ 3.0. However, in reality, cabbage can only effectively absorb... Nutrients in the place, The value at that point has almost no effect on it. Therefore, the true effective interpolation should be closer to 2.0, thus introducing a large error into the traditional Kriging interpolation method.

[0041] Based on the above scenario, this embodiment of the invention provides an improved Kriging interpolation optimization algorithm, and then constructs an adaptive Kriging spatial interpolation model to realize the interpolation of soil electrical conductivity data.

[0042] In some embodiments, adaptive kriging spatial interpolation is performed on the soil electrical conductivity data included in the aligned multimodal agricultural data to obtain a farmland state data stream. This can be achieved in the following ways, which are described in detail below.

[0043] First, an adaptive kriging spatial interpolation model is constructed based on the root density function of the target crop. The adaptive kriging spatial interpolation model is then used to perform adaptive kriging spatial interpolation on the soil electrical conductivity data included in the growth environment data to obtain the corresponding soil interpolation data.

[0044] Here, the root density function is determined by the crop variety of the target crop, denoted as . Specifically, it is expressed as follows: (2) Where r represents the root influence radius of the target crop, for example, the root influence radius of cabbage is 20cm. This indicates known sampling points for soil electrical conductivity data, which are included in the growth environment data. This represents the target interpolation point, i.e., the sampling point that needs to be interpolated.

[0045] Root density function The influence domain of the target crop root system is defined, and this function quantifies the area located in... The soil is located in The actual contribution potential of the crop is determined by the distance between the two. The contribution decreases exponentially with increasing linear distance. The root density function can perceive anisotropy, as root distribution is naturally anisotropic (e.g., extending downwards and outwards differently), and the root density function naturally describes this directional effect. Traditional Kriging is isotropic and cannot handle this directional difference. By introducing the root density function... This transforms the interpolation method from geometric space to the biological space perceived by crops.

[0046] Furthermore, an adaptive Kriging space interpolation model is constructed based on the root density function. Here, this embodiment of the invention proposes a biological weight as the weight coefficient of the interpolation algorithm to construct the adaptive Kriging space interpolation model. The interpolation weights of the traditional Kriging interpolation algorithm... The geometrical weight represents the confidence level, while the biological weight represents the importance of the target crop in the biological space, denoted as... · Interpolation based on biological weights can first suppress invalid sampling points. For sampling points outside the root system, even if they are geometrically close, The values ​​are also extremely small, leading to a significant reduction in biological weights. Traditional methods would assign excessively high weights to these sampling points, contaminating the interpolation results. Secondly, it can enhance the effectiveness of sampling points; for sampling points within the root system, their... The values ​​are large, thus increasing the biological weight and making it more decisive for the results. Traditional methods do not fully understand these sampling points and fail to make full use of their information.

[0047] In some embodiments, the adaptive kriging space interpolation model is represented by the following formula: (3) in, This represents the root density function of the target crop. Indicates the target interpolation point. This indicates the known sampling points for soil electrical conductivity data, which are included in the growth environment data. This represents the soil interpolation data obtained after interpolating the target interpolation point. The biological weights are represented by , and n represents the number of target interpolation points. This represents the soil electrical conductivity data corresponding to the known sampling points. This represents the j-th target interpolation point.

[0048] The adaptive Kriging spatial interpolation model proposed in this invention fully utilizes the biological laws of the target crop's root distribution by introducing a root density function. It can perform interpolation even when the root system of known sampling points of the target crop affects the target interpolation root system. In addition, by constructing biological weights, it effectively suppresses the information of invalid sampling points and enhances the information of valid sampling points, thereby achieving accurate interpolation of soil electrical conductivity data. Compared with the traditional Kriging spatial interpolation algorithm, it can effectively reduce interpolation errors, especially in dense root areas of the target crop, where its interpolation accuracy is significantly improved.

[0049] Next, adaptive kriging spatial interpolation is performed on the soil electrical conductivity data, which is included in the growth environment data, to obtain the corresponding soil interpolation data.

[0050] Here, we first determine the known sampling points for soil electrical conductivity data, which are included in the growth environment data. Then, the soil conductivity data of these known sampling points are input one by one into the adaptive Kriging space interpolation model of the above formula (3) for calculation. Finally, the model outputs the target crop at each target interpolation point. Soil interpolation data at the location.

[0051] After adaptive kriging spatial interpolation of soil electrical conductivity data, spatial resolution is unified for multimodal agricultural data, including soil interpolation data. This is achieved through multi-source data gridding. Specifically, hyperspectral imagery (such as NDVI vegetation index), agricultural machinery trajectories (such as operation path coverage), and the aforementioned soil data after adaptive kriging interpolation are uniformly resampled to the same spatial grid (e.g., 10cm×10cm), thereby ensuring consistent spatial resolution.

[0052] Furthermore, time series alignment is performed on the data after the spatial resolution is unified. Since the sampling frequency of each data is different, the time series alignment here is to perform high-frequency prediction on the low-frequency data involved. Specifically, high-frequency prediction is achieved by constructing a multimodal time series compensation function.

[0053] First, the correlation gradient between soil parameters and environmental variables in the growth environment data is determined. Based on the correlation gradient, a multimodal time series compensation function is constructed. Then, the multimodal time series compensation function is called to perform high-frequency prediction on low-frequency values ​​in the growth environment data after spatial resolution is unified, so as to obtain high-frequency predicted environmental data.

[0054] Here, the soil parameters in the growth environment data are soil interpolation data after adaptive Kriging spatial interpolation and soil pH value, while the environmental variables are soil temperature and soil moisture. The correlation gradient is used to characterize the relationship between soil electrical conductivity, soil pH value, and soil temperature and moisture, and is expressed as follows: S represents soil parameters, and E represents environmental variables.

[0055] Further based on correlation gradient Construct a multimodal timing compensation function, expressed as: (4) in, and This represents the time points of the k-th and k+1-th soil samplings. This represents the real-time rate of environmental change E. This is the associated gradient. This represents the low-frequency soil parameters collected at the time point of the k-th sampling. This represents high-frequency predicted environmental data, that is, soil parameters after high-frequency prediction.

[0056] Furthermore, for the low-frequency values ​​in the growth environment data after the spatial resolution is unified, a multimodal time series compensation function is used to perform high-frequency prediction of the low-frequency values ​​in the growth environment data, thus obtaining high-frequency predicted environment data.

[0057] The following describes the specific process of high-frequency prediction.

[0058] First, the integration interval is divided into N small steps according to the above formula (4). , For example, we can take 1 second, so we have: (5) Each small step prediction is based on the prediction value of the previous step. and real-time environmental change rate This is used for prediction. Therefore, when new soil parameters... Upon arrival, calibrate the associated gradient. , is represented as: (6) Where 'a' represents the learning rate, with a value of 0.1. This represents the soil parameters at the (k+1)th soil sampling point. This represents the high-frequency predicted environmental data of soil parameters at the time point of the (k+1)th soil sampling. This represents the cumulative change of environmental variable E between the time point of the kth soil sampling and the time point of the (k+1)th soil sampling.

[0059] Therefore, using the above formulas (5) and (6), between the time point of the kth soil sampling and the time point of the (k+1)th soil sampling, through N small steps... By adding N new soil parameters, the collection frequency of soil parameters is increased, enabling high-frequency prediction of low-frequency values ​​and obtaining high-frequency predicted environmental data.

[0060] Finally, a farmland status data stream was constructed based on soil interpolation data, high-frequency predicted environmental data, and crop growth data and agricultural machinery operation data after spatial resolution unification.

[0061] Here, a soil electrical conductivity distribution map with a resolution of 10cm×10cm can be determined based on soil interpolation data and synchronized with the image timestamp of the hyperspectral image corresponding to the crop growth data. After unifying the spatial resolution, the crop growth data and agricultural machinery operation data undergo pixel-level spatial matching between the agricultural machinery trajectory data and the hyperspectral image to generate spatial matching results, ensuring pixel consistency between the agricultural machinery trajectory data and the hyperspectral image. The soil electrical conductivity distribution map and the spatial matching results are then merged to output a unified spatiotemporally aligned data stream, which is then fused with high-frequency predicted environmental data to form a farmland status data stream.

[0062] In this embodiment of the invention, soil electrical conductivity is interpolated and supplemented using Kriging interpolation. Compared with traditional methods, this can reduce interpolation errors. In areas with dense root systems of the target crop, the interpolation accuracy is significantly improved. Based on this, after unifying the spatial resolution, a multimodal temporal compensation function is used to align the time series of data, increase the data acquisition frequency, effectively fill data gaps, and effectively reduce the deployment density of soil sensors. Full farmland coverage can be achieved with only a small number of sampling points and high-frequency environmental data, providing a unified data foundation for subsequent operational decisions.

[0063] Step 103: Determine the weight of each feature data in the multimodal feature data according to the growth stage index of the target crop, obtain the multimodal weight, and determine the multimodal feature vector based on the multimodal weight.

[0064] The multimodal feature data here is extracted from farmland status data streams. It includes feature data in three modalities: environmental features, crop physiological features, and agricultural machinery features. Environmental features specifically include normalized temperature and humidity, light intensity, and soil electrical conductivity, with a feature dimension of 5. Crop physiological features are normalized vegetation index (NVDI), chlorophyll content, and leaf water content extracted from hyperspectral imagery, with a feature dimension of 3. Agricultural machinery features include agricultural machinery operating speed, spraying flow rate, and navigation path deviation, also with a feature dimension of 3.

[0065] In some embodiments, the weights of each feature data in the multimodal feature data are determined based on the growth stage index of the target crop to obtain the multimodal weights. This can be achieved in the following ways, which are explained in detail below.

[0066] First, based on the environmental and physiological data corresponding to the target crop, the growth stage index of the target crop is determined.

[0067] The growth stage index of the target crop is predefined and denoted as . The value ranges from 0 to 1 and is constructed based on the accumulated temperature model and morphological characteristics of the target crop. In practice, it is calculated based on the environmental and physiological data corresponding to the target crop. The environmental data is the optimal temperature for the target crop's growth, and the physiological data is the leaf area index of the target crop. Taking cabbage as an example, the target crop is... It is expressed as follows: (7) in, This represents the sigmoid normalization function. Indicates the daily average temperature. This indicates the optimal temperature for cabbage growth. and This represents an empirical coefficient, and the empirical coefficient is set differently for different target crops, such as cabbage. Take 0.8, Let 0.2 be the value, and t represent the growth cycle of the cabbage. Indicates the initial time. This represents the leaf area index of cabbage within a growth cycle t.

[0068] Furthermore, based on the growth stage index, the environmental weights of environmental characteristics, the physiological weights of crop physiological characteristics, and the operational weights of agricultural machinery characteristics are dynamically allocated to obtain multimodal weights.

[0069] Here, a multimodal attention function is designed to input the corresponding growth stage index. The input is fed into a multimodal attention function, which outputs the weights corresponding to each feature data, denoted as . The calculation process of the multimodal attention function is as follows: (8) Where m represents the feature data of the m-th mode. This represents the total number of modalities in the feature data. The feature data in multimodal feature data includes feature data from three modalities: environmental features, crop physiological features, and agricultural machinery features. It is 3. The modal sensitivity coefficient represents the modal sensitivity coefficient, which varies at different growth stages (t) of the target crop. For example, the modal sensitivity coefficient for environmental characteristics is 2.0 during the seedling stage of cabbage, while it is 3.0 during the harvest stage of cabbage for crop physiological characteristics.

[0070] In the early stages of the target crop's growth cycle, such as the seedling stage, environmental data may be more important, and therefore environmental characteristics can be assigned a higher weight. In the middle stages of the target crop's growth cycle, such as the growing season, crop physiological data such as chlorophyll content and normalized vegetation index are more important, so crop physiological characteristics should be assigned a better weight. In the later stages of the target crop's growth cycle, such as the harvest period, agricultural machinery operation data may be more helpful for yield prediction, so agricultural machinery characteristics should be assigned a higher weight.

[0071] This invention proposes a growth stage-sensitive attention mechanism to dynamically adjust the weights of various feature data, thus solving the problem that traditional fixed weights cannot adapt to the decision-making needs of crops at different growth stages.

[0072] Next, the multimodal feature vector is determined based on the multimodal weights.

[0073] First, each feature data in the multimodal feature data is input into a multilayer perceptron to obtain the output features. There are multiple multilayer perceptrons, and each multilayer perceptron processes one feature data independently.

[0074] Furthermore, the output features are weighted and fused according to the multimodal weights to obtain a multimodal feature vector, represented as follows: The formula is as follows: (9) in, This represents the multimodal weights, namely the environmental weights of environmental characteristics, the physiological weights of crop physiological characteristics, and the operational weights of agricultural machinery characteristics. This represents the feature data of the m-th modality. MLP represents the processing function of the multilayer perceptron.

[0075] Weighted fusion yields multimodal feature vectors This information can then be used for subsequent decision-making.

[0076] In this embodiment of the invention, multimodal weighting is used to achieve weighted fusion of feature data. Each modality is designed with an independent multilayer perceptron to process the feature data, avoiding feature confusion. Compared with fixed-weight feature fusion, multimodal data representation achieves higher accuracy.

[0077] Step 104: Input the multimodal feature vector, historical operation data of the target crop, and future environmental prediction results into the operation decision model to obtain the operation decision scheme for the target crop.

[0078] The operational decision-making model here can be modeled using a Dynamic Bayesian Network (DBN). Furthermore, it requires obtaining historical operational data for the target crop and future environmental predictions to aid in decision-making. During the decision-making process, a multi-objective optimization function is constructed, for example, to maximize crop yield or minimize resource consumption, to solve for the optimal operational strategy. Then, the Dynamic Bayesian Network is used to generate corresponding operational decision plans, serving as the operational decision scheme for the target crop to control the agricultural machinery for automated operation.

[0079] In some embodiments, before inputting the multimodal feature vectors, historical operation data of the target crop, and future environmental prediction results into the operation decision model, it is also necessary to obtain historical operation data and future environmental prediction results.

[0080] Specifically, the first step is to determine the historical agricultural data relationships and historical operation records as the historical operation data of the target crop. The historical agricultural data relationships include the relationship between spraying amount and crop yield. Here, we can collect the historical pesticide spraying amount data and the corresponding crop yield data of the target crop to form a spraying amount-yield relationship table. The historical operation records include agricultural machinery failure records, that is, records of failures that occurred in agricultural machinery during operation.

[0081] In making operational decisions, this invention incorporates the relationship between spraying amount and crop yield, as well as future environmental forecasts, to ensure that the operational decision-making model can make effective decisions under the influence of future environmental factors and to maximize crop yield. It can also dynamically adjust decisions based on historical experience and future forecasts, thereby improving the effectiveness of operational decisions.

[0082] In some embodiments, the multimodal feature vector, historical operation data of the target crop, and future environmental prediction results are input into the operation decision model to obtain the operation decision scheme for the target crop. This can be achieved in the following ways, which are described in detail below.

[0083] A job decision model is constructed based on short-term and long-term nodes of a dynamic Bayesian network.

[0084] Here, the job decision model is obtained through Dynamic Bayesian Network (DBN). The DBN network structure adopts a time-series hierarchical design, defining two layers of dynamic nodes, including short-term nodes and long-term nodes, as follows: (10) in, It represents high-frequency states on a second-by-second basis, such as soil moisture and agricultural machinery position, and updates them through independent Markov chains to reduce long-term noise interference. This indicates a daily-level prediction status, such as the future. Hourly crop yield and resource consumption, f represents the integration of short-term, second-level high-frequency states within a historical window. The cumulative effect. This indicates the cross-layer weight, such as the influence coefficient of current soil moisture on the future yield of the target crop. The historical window is used to calculate the predicted duration.

[0085] Short-run nodes (time t) represent environmental conditions (temperature, humidity, light intensity), crop physiological indicators (chlorophyll content), and agricultural machinery parameters (operating speed). Their role in agricultural decision-making is to control the immediate actions of agricultural machinery (such as obstacle avoidance steering, spraying start and stop). Long-run nodes (t∼t+ΔT) represent crop yield forecasts, cumulative resource consumption, and equipment wear and tear. Their role in agricultural decision-making is to optimize long-term resource allocation, such as planning harvesting routes 48 hours in advance.

[0086] Furthermore, this embodiment of the invention introduces a Conditional Probability Table (CPT) to dynamically update the nodes in the dynamic Bayesian network. By introducing an environment-sensitive factor, the dependencies between nodes are adjusted, as shown in Equation 11. (11) in, The baseline probability, specifically a matrix, is based on the state transition patterns statistically derived from historical data, such as the probability of transitioning from healthy to diseased / pested crops. This can reflect the long-term growth patterns of crops, such as the probability of downy mildew in cabbage seedlings. The driving probability is specifically a matrix that represents the conditional probability driven by the environment, such as the impact of real-time environment (temperature, humidity, light) on health status (e.g., pests and diseases), and capturing the instantaneous impact of sudden weather on crops, such as high temperature inducing sunscald. The value represents the degree of environmental abrupt change, ranging from [0, 1], and is determined based on the real-time environment. For example, in the event of a sudden heavy rain, then... The automatic decrease tends towards 0, which is represented as: (12) Where k represents the environmental sensitivity coefficient, which defaults to 0.5 and is used to switch the steepness. The larger k is, the easier it is to trigger a response update. For example, when k=1, the environmental sensitivity is the maximum, and the weights of the baseline probability and the driving probability will be updated when the temperature drops by 2 degrees Celsius. Represented as a sliding environment value, specifically a vector, it is used to measure real-world abrupt changes in the environment, such as heavy rain or strong winds. This represents the real-time environmental state observation value at time t, including multi-dimensional environmental variables, namely [temperature, humidity, wind speed], such as [28.5℃, 65%, 3.2m / s].

[0087] When making job decisions, the input to the dynamic Bayesian network is a multimodal feature vector. Historical operational data of the target crop and future environmental predictions.

[0088] During real-time inference of the dynamic Bayesian network, multimodal feature vectors, historical operation data of the target crop, and future environmental prediction results are input into the operation decision model as particles. The particle swarm algorithm is then called to update the particles, thereby updating the short-term and long-term nodes and obtaining the operation decision scheme for the target crop.

[0089] The particle weights are updated based on a time decay factor, expressed as: (13) in, This represents the weight of the particle at time t-1. This represents the weight of the particle at time t. Represents the time decay factor, when A value of 0.1 indicates that the particle weight decays by 63% after 10 seconds. Indicates the most recent observation timestamp, Represents a node The probability distribution value determined under the conditional probability table.

[0090] During the process of updating particles using the particle swarm optimization algorithm, the particle weights are updated to update both short-term and long-term nodes. When a node is updated, the dependencies between nodes are used to update other nodes. Finally, after the iteration update is completed, the optimal set of operation decision instructions is determined based on the updated nodes, which serves as the operation decision scheme for the target crop.

[0091] In this embodiment of the invention, the particle weights are updated through time decay when making job decisions. During node updates, environmental sensitivity factors are used to adjust the dependencies between nodes, so that job decision updates no longer depend on historical job data, solving the problem of excessive reliance on historical data in traditional particle filtering. Furthermore, triggering node updates through environmental sensitivity factors can improve the response speed to sudden environmental events during job decisions.

[0092] The present invention provides an agricultural operation decision-making method and apparatus based on multimodal fusion. This method unifies crop growth data, growth environment data, and agricultural machinery operation data through unified alignment processing. It also employs adaptive Kriging space interpolation processing for the included soil conductivity data, facilitating the unification of multimodal agricultural data into a single spatiotemporal coordinate system and enabling multi-dimensional data fusion to address the problem of data fragmentation. Furthermore, by determining the feature weights of the multi-source data using the growth stage index of the target crop, the fusion processing of various feature data is achieved, enabling an accurate representation of the current farmland state. Finally, based on the multimodal features, and combined with historical operation data and future environmental conditions, operation decisions are made. This ensures that the operation decision scheme adapts to the growth stage of the target crop and the future environment, improving the effectiveness and accuracy of agricultural machinery operation.

[0093] The following describes the job decision-making device based on multimodal fusion provided by the present invention. The job decision-making device based on multimodal fusion described below can be referred to in correspondence with the job decision-making method based on multimodal fusion described above.

[0094] like Figure 2 As shown, the operation decision-making device based on multimodal fusion includes: an acquisition module 201, an alignment module 202, a determination module 203, and a decision module 204.

[0095] Specifically, the acquisition module 201 is used to acquire multimodal agricultural data to be processed, including crop growth data, growth environment data, and agricultural machinery operation data; the alignment module 202 is used to align the multimodal agricultural data and perform adaptive Kriging space interpolation on the soil conductivity data included in the aligned multimodal agricultural data to obtain a farmland state data stream; the determination module 203 is used to determine the weight of each feature data in the multimodal feature data according to the growth stage index of the target crop to obtain multimodal weights, and determine the multimodal feature vector based on the multimodal weights, wherein the multimodal feature data is extracted from the farmland state data stream; and the decision module 204 is used to input the multimodal feature vector, the historical operation data of the target crop, and the future environmental prediction results into the operation decision model to obtain the operation decision scheme for the target crop.

[0096] In some embodiments, the acquisition module 201 is further configured to determine historical agricultural data relationships and historical operation records as historical operation data of the target crop, wherein the historical agricultural data relationships include the relationship between spraying amount and crop yield, and the historical operation records include agricultural machinery malfunction records; The future meteorological information of the current growth environment of the target crop is determined as the future environmental prediction result. The future meteorological information includes the future rainfall probability and future wind speed change information of the current growth environment.

[0097] It should be noted that the beneficial effects of the multimodal fusion-based job decision-making device described here can be compared with the beneficial effects of the multimodal fusion-based job decision-making method mentioned above. Therefore, the effective effects of the multimodal fusion-based job decision-making device will not be elaborated here.

[0098] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other through the communications bus 340. The processor 310 can call logical instructions in the memory 330 to execute a multimodal fusion-based job decision method. This method includes: acquiring multimodal agricultural data to be processed, the multimodal agricultural data including crop growth data, growth environment data, and agricultural machinery operation data; aligning the multimodal agricultural data and performing adaptive kriging space interpolation on the soil conductivity data included in the aligned multimodal agricultural data to obtain a farmland state data stream; determining the weights of each feature data in the multimodal feature data based on the growth stage index of the target crop to obtain multimodal weights; determining a multimodal feature vector based on the multimodal weights, the multimodal feature data being extracted from the farmland state data stream; and inputting the multimodal feature vector, historical operation data of the target crop, and future environmental prediction results into a job decision model to obtain a job decision scheme for the target crop.

[0099] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal fusion-based job decision method provided by the above methods. The method includes: acquiring multimodal agricultural data to be processed, the multimodal agricultural data including crop growth data, growth environment data, and agricultural machinery operation data; aligning the multimodal agricultural data and performing adaptive Kriging space interpolation on the soil conductivity data included in the aligned multimodal agricultural data to obtain a farmland state data stream; determining the weight of each feature data in the multimodal feature data according to the growth stage index of the target crop to obtain multimodal weights; determining a multimodal feature vector based on the multimodal weights, the multimodal feature data being extracted from the farmland state data stream; and inputting the multimodal feature vector, the historical operation data of the target crop, and the future environmental prediction results into a job decision model to obtain a job decision scheme for the target crop.

[0101] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the operation decision method based on multimodal fusion provided by the above methods. This method includes: acquiring multimodal agricultural data to be processed, the multimodal agricultural data including crop growth data, growth environment data, and agricultural machinery operation data; aligning the multimodal agricultural data and performing adaptive kriging space interpolation on the soil conductivity data included in the aligned multimodal agricultural data to obtain a farmland state data stream; determining the weights of each feature data in the multimodal feature data according to the growth stage index of the target crop to obtain multimodal weights; determining a multimodal feature vector based on the multimodal weights, the multimodal feature data being extracted from the farmland state data stream; and inputting the multimodal feature vector, historical operation data of the target crop, and future environmental prediction results into an operation decision model to obtain an operation decision scheme for the target crop.

[0102] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A job decision-making method based on multimodal fusion, characterized in that, include: Acquire multimodal agricultural data to be processed, including crop growth data, growth environment data, and agricultural machinery operation data; The multimodal agricultural data is aligned, and adaptive Kriging spatial interpolation is performed on the soil electrical conductivity data included in the aligned multimodal agricultural data to obtain a farmland state data stream. The weights of each feature data in the multimodal feature data are determined based on the growth stage index of the target crop to obtain the multimodal weights. The multimodal feature vectors are then determined based on the multimodal weights. The multimodal feature data is extracted from the farmland state data stream. The multimodal feature vector, the historical operation data of the target crop, and the future environmental prediction results are input into the operation decision model to obtain the operation decision scheme for the target crop.

2. The job decision-making method based on multimodal fusion according to claim 1, characterized in that, The multimodal agricultural data is aligned in the following manner: Using the second pulse signal of the agricultural machinery operation data as a reference, the timestamps of the crop growth data and the growth environment data are clocked for synchronization. Based on the time of data collection of the crop growth data, the growth environment data and the agricultural machinery operation data are interpolated simultaneously. Align the crop growth data, the growth environment data, and the agricultural machinery operation data in a spatial coordinate system.

3. The job decision-making method based on multimodal fusion according to claim 1, characterized in that, The adaptive Kriging spatial interpolation processing is performed on the soil electrical conductivity data included in the aligned multimodal agricultural data to obtain a farmland state data stream, including: An adaptive kriging spatial interpolation model is constructed based on the root density function of the target crop. The adaptive kriging spatial interpolation model is then used to perform adaptive kriging spatial interpolation on the soil electrical conductivity data included in the growth environment data to obtain the corresponding soil interpolation data. Spatial resolution is unified for multimodal agricultural data including the soil interpolation data; Determine the correlation gradient between soil parameters and environmental variables in the growth environment data, and construct a multimodal time-series compensation function based on the correlation gradient; The multimodal temporal compensation function is invoked to perform high-frequency prediction on low-frequency values ​​in the growth environment data after spatial resolution unification, thereby obtaining high-frequency predicted environment data; A farmland status data stream is constructed based on the soil interpolation data, the high-frequency predicted environmental data, the crop growth data after spatial resolution unification, and the agricultural machinery operation data.

4. The job decision-making method based on multimodal fusion according to claim 3, characterized in that, The adaptive Kriging space interpolation model is expressed by the following formula: in, This represents the root density function of the target crop. Indicates the target interpolation point. This indicates the known sampling points for soil electrical conductivity data, which are included in the growth environment data. This represents the soil interpolation data obtained after interpolating the target interpolation point. The weights represent the weights of the Kriging space interpolation algorithm, and n represents the number of target interpolation points. This represents the soil electrical conductivity data corresponding to the known sampling points. This represents the j-th target interpolation point.

5. The job decision-making method based on multimodal fusion according to claim 1, characterized in that, The multimodal feature data includes environmental features, crop physiological features, and agricultural machinery features. The weights of each feature data in the multimodal feature data are determined based on the growth stage index of the target crop, resulting in multimodal weights, including: Based on the environmental and physiological data corresponding to the target crop, the growth stage index of the target crop is determined; The environmental weights of the environmental features, the physiological weights of the crop physiological features, and the operational weights of the agricultural machinery features are dynamically allocated based on the growth stage index to obtain multimodal weights.

6. The job decision-making method based on multimodal fusion according to claim 1, characterized in that, The determination of the multimodal feature vector based on the multimodal weights includes: Each feature data in the multimodal feature data is input into a multilayer perceptron to obtain the output features; The output features are weighted and fused according to the multimodal weights to obtain a multimodal feature vector.

7. The job decision-making method based on multimodal fusion according to claim 1, characterized in that, Before inputting the multimodal feature vector, the historical operation data of the target crop, and the future environmental prediction results into the operation decision model, the method further includes: The historical agricultural data relationships and historical operation records are determined as the historical operation data of the target crop. The historical agricultural data relationships include the relationship between spraying amount and crop yield, and the historical operation records include agricultural machinery failure records. The future meteorological information of the current growth environment of the target crop is determined as the future environmental prediction result. The future meteorological information includes the future rainfall probability and future wind speed change information of the current growth environment.

8. The job decision-making method based on multimodal fusion according to claim 7, characterized in that, The step of inputting the multimodal feature vector, the historical operation data of the target crop, and the future environmental prediction results into the operation decision model to obtain the operation decision scheme for the target crop includes: A job decision model is constructed based on short-term and long-term nodes of a dynamic Bayesian network. The multimodal feature vector, historical operation data of the target crop, and future environmental prediction results are input as particles into the operation decision model. The particle swarm optimization algorithm is called to update the particles to update the short-term nodes and the long-term nodes, thereby obtaining the operation decision scheme for the target crop. The weight of the particles is updated based on the time decay factor.

9. A job decision-making device based on multimodal fusion, characterized in that, include: The acquisition module is used to acquire multimodal agricultural data to be processed, including crop growth data, growth environment data, and agricultural machinery operation data. The alignment module is used to align the multimodal agricultural data and perform adaptive Kriging space interpolation on the soil electrical conductivity data included in the aligned multimodal agricultural data to obtain a farmland state data stream. The determination module is used to determine the weight of each feature data in the multimodal feature data according to the growth stage index of the target crop, to obtain the multimodal weight, and to determine the multimodal feature vector based on the multimodal weight. The multimodal feature data is extracted from the farmland state data stream. The decision module is used to input the multimodal feature vector, the historical operation data of the target crop, and the future environmental prediction results into the operation decision model to obtain the operation decision scheme of the target crop.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the job decision method based on multimodal fusion as described in any one of claims 1 to 8.

Images