Digital rural intelligent management system based on multi-modal data fusion

Abstract

The invention belongs to the technical field of digital country intelligent management systems, and particularly relates to a digital country intelligent management system based on multi-modal data fusion, comprising a heterogeneous sensor network which is deployed in farmland and other places, and adopts a region division algorithm based on a Voronoi diagram and an improved ant colony algorithm to plan an unmanned aerial vehicle inspection path through LoRaWAN protocol networking; the edge computing node is connected with the heterogeneous sensor network, an FPGA preprocessing module is arranged in the edge computing node, and a hybrid noise reduction algorithm based on wavelet transform-Kalman filtering is configured; the central server is connected with an edge computing node through a hierarchical data bus, and comprises a resource allocation optimization module, a three-dimensional visualization engine and the like; the intelligent interaction terminal supports multi-mode man-machine interaction. In addition, the system further comprises an anomaly detection module, and an LSTM-AE self-encoder is constructed to achieve anomaly detection and graded alarm. According to the system, intelligent management and multi-modal data fusion processing of the digital country are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital village intelligent management systems, and in particular relates to a digital village intelligent management system based on multimodal data fusion. Background Art

[0002] Existing sensor networks often employ single-modality deployment (such as meteorological or soil monitoring only), lacking deep fusion of heterogeneous data and limiting the dimensionality of environmental perception. Furthermore, sensor deployment in rural settings often ignores differences in terrain and crop growth dynamics, resulting in rigid deployment density and path planning strategies and uneven data coverage. In data processing, raw signals are severely affected by the complex electromagnetic environment of farmland. Traditional filtering algorithms (such as single Kalman filters) struggle to balance noise suppression with effective signal preservation. Furthermore, edge computing resources are limited, resulting in insufficient real-time performance. Resource allocation models often employ single-objective optimization, failing to balance the multi-objective conflicts of cost and supply-demand balance. Three-dimensional visualization often relies on independent GIS or BIM platforms, resulting in low model accuracy and difficulty integrating multi-source data. Furthermore, rural user interaction relies on traditional terminals that lack dialect adaptation and multimodal interaction capabilities, limiting system universality. At the communication level, data bus transmission cycles are fixed, making it difficult to dynamically adapt to changes in node size, which can easily lead to congestion or idle resources. Anomaly detection often relies on threshold rules, lacking the ability to recognize implicit patterns in time series data. The above problems urgently require a systematic solution that integrates multimodal fusion, adaptive optimization, and intelligent interaction capabilities. Summary of the Invention

[0003] In response to the technical problems existing in the above-mentioned traditional rural management system, the present invention provides a digital rural intelligent management system based on multimodal data fusion.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A digital village intelligent management system based on multimodal data fusion, characterized by comprising:

[0006] Heterogeneous sensor networks are deployed in farmland, roads, and public facilities. They include soil composition sensors, meteorological sensors, video acquisition equipment, and drone inspection modules. Each sensor is hierarchically networked using the LoRaWAN protocol.

[0007] An edge computing node, connected to a heterogeneous sensor network, having a built-in FPGA preprocessing module and a hybrid noise reduction algorithm based on wavelet transform and Kalman filtering;

[0008] The central server is connected to the edge computing node through a hierarchical data bus. The hierarchical data bus adopts a time-triggered communication mechanism with a transmission period of T=min{2n T0|n∈N}, where T0 is the reference clock period and T is the data bus transmission period; the central server includes:

[0009] Resource allocation optimization module, which uses the improved NSGA-II algorithm to achieve multi-objective optimization of hydropower resources;

[0010] 3D visualization engine, building a digital twin village model based on BIM-GIS fusion modeling technology;

[0011] An intelligent interactive terminal is connected to a central server and is equipped with a speech recognition module to support multimodal human-computer interaction.

[0012] The deployment method of the heterogeneous sensor network includes:

[0013] Based on the Voronoi diagram region partitioning algorithm, the sensor layout density is ρ = K·ln(S·δ+1), where: K is a proportional constant used to calibrate the combined effects of terrain and crop factors on density; S is the terrain complexity coefficient, and δ is the crop growth index;

[0014] The UAV inspection path is planned using the improved ant colony algorithm, and the pheromone update rule is:

[0015]

[0016] Where: τ ij (t) is the pheromone concentration from node i to node j in the UAV inspection path at time t; τ ij (t+1) is the updated pheromone concentration on edge (i, j) at time (t+1), that is, after completing one inspection iteration; is the pheromone increment of the optimal path on the edge (i, j) in the current iteration, Get the pheromone increment of the current optimal path.

[0017] The expression of the wavelet transform-Kalman filter hybrid noise reduction algorithm in the edge computing node is:

[0018]

[0019] Where: α is the dynamic weight coefficient, Q and R are the covariance matrices of process noise and observation noise respectively, y(t) is the input original sensor signal, represents the output signal after the hybrid noise reduction algorithm is processed. DWT(·) represents discrete wavelet transform, which is used to decompose the original signal into different frequency domains and extract the noise and effective signal components. -1(·) represents the inverse discrete wavelet transform, which reconstructs the processed wavelet coefficients back into the time domain signal; Φ(·) represents the threshold processing function of the wavelet coefficients; KF(y(t)|Q,R) is the Kalman filter algorithm, the input is the original signal y(t), and the output is the filtered estimate.

[0020] The FPGA preprocessing module of the edge computing node includes:

[0021] Data check unit, using CRC-32C cyclic redundancy check code;

[0022] The feature extraction unit is equipped with a temporal feature extractor based on 1D-CNN, and the network structure is [Conv(64,k=5)―MaxPool(2)―Conv(128,k=3)―GlobalAvgPool];

[0023] Among them: Conv(64,k=5) represents the first one-dimensional convolution layer, which is used to extract the local features of the time series signal collected by the sensor; the number of output channels is 64, which means that 64 different time series features are extracted; k=5 means that the convolution kernel size is 5, that is, each convolution operation considers the signals of the current moment and the previous 4 moments, capturing the local time series dependency of 5 time steps; MaxPool(2) represents the maximum pooling layer with a pooling window size of 2, which is used to reduce the dimension and retain the maximum eigenvalue of the local area; Conv(128,k=3) represents the second one-dimensional convolution layer, with an output channel number of 128 and a convolution kernel size of 3; GlobalAvgPool is a global average pooling layer, which takes the average value of the entire time dimension and compresses the multi-dimensional feature map into a feature vector of fixed length;

[0024] Cache management unit, implements LRU-K replacement strategy.

[0025] The objective function of the resource allocation optimization module is:

[0026]

[0027] Where: x is the resource allocation decision variable vector, F(x) is the objective function vector of multi-objective optimization, which contains two sub-goals f1(x) and f2(x), f1(x) is the weighted absolute error sum, w i is the weight of the i-th demand indicator, a i is the actual assigned value of the i-th indicator, is the target demand value of the i-th indicator; f2(x) is the total cost of resource allocation, c j is the cost coefficient of the j-th type of resource, x j is the allocation amount of the jth type of resources.

[0028] The workflow of the resource allocation optimization module includes:

[0029] Establish a Pareto frontier search mechanism and use the ε-dominance strategy to maintain the diversity of the solution set;

[0030] Design the adaptive crossover probability p c =0.9-0.5·(g / G), where g is the current generation and G is the total evolution generation;

[0031] Introduce a local search operator and perform simulated annealing optimization on the non-dominated solution: T(k) = t0·0.95 k , where: T(k) is the temperature value at the kth iteration, which measures the probability of accepting an inferior solution during the search process; t0 is the initial temperature, which determines the exploration intensity in the initial stage of the algorithm; k is the number of annealing iterations (k = 0, 1, 2, ...), which represents the current step in the local search process.

[0032] The three-dimensional visualization engine includes:

[0033] In the multi-source data fusion layer, the improved ICP algorithm is used to realize point cloud registration. The objective function is:

[0034] min∑ω i ·||R·p i +t―q i || 2

[0035] Where: i is the confidence weight; p i Represents the i-th 3D coordinate point of the registered point cloud data, including the farmland point cloud scanned by UAV LiDAR and the road point cloud collected by vehicle-mounted LiDAR; q i Indicates the point cloud data of the registration benchmark and p i The corresponding nearest neighbor points include high-precision BIM model point clouds and GIS georeferenced point clouds; R represents a 3×3 orthogonal rotation matrix, which describes the rotational posture of the source point cloud relative to the target point cloud and contains 3 degrees of freedom; t represents a 3×1 translation vector, which describes the translation offset of the source point cloud relative to the target point cloud;

[0036] Real-time rendering layer, built on WebGL 2.0, supports LOD multi-level of detail model switching.

[0037] The voice recognition module of the intelligent interactive terminal includes:

[0038] The dialect adaptation layer adopts the adversarial transfer learning framework, and the loss function is:

[0039] L=L CTC +λ·L domain

[0040] Where: L is the total loss, which comprehensively measures the overall error of the speech recognition model in the speech-to-text alignment task and the domain transfer adaptation task, and guides the model to optimize the two capabilities simultaneously through weighted summation; L CTC L is a connectionist temporal classification loss, which is used in scenarios where the length of the speech feature sequence is inconsistent with the length of the text sequence; domain is the domain loss, which is used to measure the distribution difference between standard Mandarin speech data and dialect speech data, and forces the model to ignore domain-specific features through adversarial training; λ is the balance L CTC and L domain The weight of importance is λ∈[0.3,0.7];

[0041] The context understanding module integrates the BERT-Base pre-trained model and compresses the model size to 30% of the original through knowledge distillation.

[0042] The communication protocol of the hierarchical data bus includes:

[0043] The physical layer uses OFDM modulation, with the number of subcarriers N = 64 and the cyclic prefix length CP = 16;

[0044] The data link layer implements TDMA time slot allocation, with a time slot width of Δt = 10ms·(1+0.1·log2M), where M is the number of access nodes.

[0045] It also includes an anomaly detection module, which is connected to the central server. The implementation method of the anomaly detection module is as follows:

[0046] Construct an LSTM-AE autoencoder with the hidden layer dimension set to 1 / 3 of the input dimension;

[0047] Defining anomaly scores Where: x(t) is the input data vector at the current time t; is the output data vector reconstructed by the autoencoder; ||·||2 is the L2 norm, which measures the point-by-point difference between the input data and the reconstructed data; σ(t) is the statistic of the hidden layer activation value at the current moment; μ σ is the long-term mean of the hidden layer statistics; γ is the weight parameter, γ = 0.7;

[0048] When A(t)>θ, a graded alarm is triggered, where θ=Q3+1.5·IQR, where Q3 is the third quartile, representing the 75% position value of the anomaly score data set; and IQR is the interquartile range, reflecting the distribution range of the middle 50% of the data.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. This invention's efficient deployment of an intelligent sensing network combines geographic characteristics with crop growth dynamics, employing an intelligent regionalization algorithm to dynamically adjust sensor density, ensuring precise coverage of complex terrain and farmland. Drone inspection routes are automatically planned using a biomimetic optimization algorithm, significantly reducing inspection time and energy consumption, enabling highly efficient environmental monitoring.

[0051] 2. This invention upgrades edge data processing capabilities by integrating advanced signal processing technologies at the edge. By combining frequency-domain and time-domain noise reduction methods, it effectively distinguishes between noise and valid signals, significantly improving data quality. Furthermore, combined with a dedicated hardware acceleration module, it enables rapid feature extraction of farmland time-series data, ensuring real-time responsiveness.

[0052] 3. This invention achieves a multi-objective balance in resource allocation by employing an evolutionary algorithm framework to optimize hydropower resource allocation, accurately matching supply and demand while reducing operating costs and supporting dynamic multi-objective trade-offs. Furthermore, it incorporates adaptive optimization strategies and local search mechanisms to accelerate decision convergence and improve the rationality of resource allocation in complex scenarios.

[0053] 4. The present invention dynamically optimizes communication transmission and designs a flexible time slot allocation mechanism to automatically adjust the data transmission rhythm according to the number of access devices, supporting large-scale node expansion while ensuring real-time performance and avoiding network congestion.

[0054] 5. This invention provides accurate and reliable anomaly warnings. It uses a deep learning model to analyze time series data characteristics and comprehensively determines anomalies through multi-dimensional indicators, significantly reducing the probability of false alarms or missed alarms. Furthermore, a graded alarm mechanism is established to achieve rapid response based on different risk levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can, without inventive effort, derive other implementation drawings based on the provided drawings.

[0056] The structures, proportions, sizes, etc. illustrated in this specification are intended solely to complement the contents disclosed herein and to facilitate understanding and reading by persons skilled in the art. They are not intended to limit the conditions under which the present invention may be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, without affecting the efficacy and objectives of the present invention, shall remain within the scope of the technical contents disclosed herein.

[0057] Figure 1 This is a system principle block diagram of the present invention;

[0058] Figure 2 This is a principle block diagram of a heterogeneous sensor network according to the present invention;

[0059] Figure 3 This is a principle block diagram of the edge computing node of the present invention; Figure 4 This is a principle block diagram of the central server of the present invention; Figure 5 This is a principle block diagram of the intelligent interactive terminal of the present invention.

[0060] Among them: 101 is a heterogeneous sensor network, 102 is an edge computing node, 1021 is a data verification unit, 1022 is a feature extraction unit, 1023 is a cache management unit, 103 is a central server, 1031 is a resource allocation optimization module, 1032 is a three-dimensional visualization engine, 1032a is a multi-source data fusion layer, 1032b is a real-time rendering layer, 104 is an intelligent interactive terminal, 1041 is a dialect adaptation layer, 1042 is a context understanding module, 105 is a hierarchical data bus, and 106 is an anomaly detection module. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of this application, not all the embodiments. These descriptions are only to further illustrate the features and advantages of the present invention, rather than to limit the claims of the present invention. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0062] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following embodiments are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0063] In the description of this application, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in this application based on the specific circumstances.

[0064] like Figure 1As shown, the system includes a heterogeneous sensor network 101, an edge computing node 102, a central server 103, an intelligent interactive terminal 104, a hierarchical data bus 105, and an anomaly detection module 106. Each module works together and is specifically implemented as follows:

[0065] 1. Heterogeneous Sensor Networks 101

[0066] Deployment structure: such as Figure 2 As shown in the figure, the sensor network covers farmland, roads and public facilities, and includes soil composition sensors (monitoring pH value and humidity), meteorological sensors (temperature, humidity, wind speed), video acquisition equipment (1080P camera) and drone inspection modules (equipped with multispectral cameras).

[0067] Communication Mechanism: Adopting the LoRaWAN protocol for hierarchical networking, nodes are divided into terminal nodes (data collection), gateway nodes (data aggregation), and relay nodes (signal enhancement). The network topology is dynamically optimized using an adaptive routing algorithm.

[0068] UAV inspection path planning: Based on the improved ant colony algorithm, the pheromone volatility factor ρ = 0.1 is set, and the pheromone increment (L best The drone performs periodic inspections along the planned route and transmits real-time data on crop growth, pests and diseases.

[0069] For example: in an area of 10km 2 For farmland areas, the terrain complexity coefficient S = 1.5, the crop growth index δ = 0.8, and the proportional constant K = 0.2, then the sensor layout density ρ = 0.2 × ln (1.5 × 0.8 + 1) = 0.18 sensors / 100m 2 After dividing the area by the Voronoi diagram, soil sensors and weather stations are deployed according to density.

[0070] 2. Edge computing node 102, such as Figure 3 As shown, it includes the following modules:

[0071] Hardware configuration: Xilinx Zynq UltraScale+ MPSoC FPGA, integrating ARM processor and programmable logic unit.

[0072] FPGA preprocessing module

[0073] Data check unit 1021: Uses CRC-32C cyclic redundancy check to check received data to ensure that no errors occurred during data transmission. CRC-32C is an efficient check algorithm that can detect sudden and random errors in data.

[0074] Feature extraction unit 1022: equipped with a 1D-CNN-based temporal feature extractor, the network structure is [Conv(64, k=5)-MaxPool(2)-Conv(128, k=3)-GlobalAvgPool]:

[0075] Conv(64, k=5): The first one-dimensional convolutional layer extracts local features from time series signals collected by sensors. The number of output channels is 64, indicating the extraction of 64 different time series features. k=5 indicates a convolution kernel size of 5, meaning that each convolution operation considers the signals at the current moment and the previous four moments, capturing local temporal dependencies across five time steps. For example, for time series data on soil moisture, this convolutional layer can extract short-term trends.

[0076] MaxPool(2): A maximum pooling layer with a pooling window size of 2, used to reduce dimensionality and retain the maximum eigenvalue of the local area, reducing the amount of data while retaining important features.

[0077] Conv(128,k=3): The second one-dimensional convolution layer has 128 output channels and a convolution kernel size of 3 to further extract more complex temporal features.

[0078] GlobalAvgPool: Global average pooling layer, which takes the average value of the entire time dimension and compresses the multi-dimensional feature map into a feature vector of fixed length to facilitate subsequent data analysis and processing.

[0079] Cache Management Unit 1023: Implements the LRU-K replacement strategy, which manages cache space based on data access history and frequency, improving data access efficiency. The LRU-K strategy records the last K access times of data to more accurately predict data usage frequency and avoid frequent cache replacements.

[0080] Hybrid noise reduction algorithm: The original signal y(t) is processed by wavelet transform-Kalman filter fusion. The dynamic weight coefficient α is dynamically adjusted according to the signal-to-noise ratio: when SNR<10dB, α=0.7 (focus on wavelet transform), when SNR≥20dB, α=0.3 (focus on Kalman filter). The process noise covariance matrix Q=diag(0.01 2 ,0.02 2 ), observation noise covariance matrix R = 0.05 2 .

[0081] For example, soil moisture sensor signals are pre-processed on the FPGA and subjected to CRC-32C check before feature extraction using a 1D-CNN. The input signal sequence length is 128 characters. After two layers of convolution and pooling, a 128-dimensional feature vector is generated for subsequent resource optimization.

[0082] 3. Central server 103, such as Figure 4 As shown, it includes the following modules:

[0083] Resource allocation optimization module 1031

[0084] Objective function: Taking the allocation of hydropower resources as an example, let the demand index i = 5 (such as irrigation water volume, power load, etc.), the objective function Weight w i Determined by entropy weight method; f2(x)=∑c j ·x j , cost coefficient c j Taken from the resource price list.

[0085] Improved NSGA-II algorithm:

[0086] Population initialization: decision variable x j Normalized in the interval [0,1], the population size is 200.

[0087] Adaptive crossover: initial crossover probability p c =0.9, and decreases linearly with the evolutionary generation g to 0.4 (total generation G = 100).

[0088] Simulated annealing local search: annealing optimization is applied to the non-dominated solution with an initial temperature t0 = 100, a temperature decay coefficient of 0.95, and 10 iterations.

[0089] For example: A village needs to allocate 2000m3 of water per day. 3 , target demand (Unit: m 3 After NSGA-II optimization, the Pareto solution set is obtained, and the solution with the lowest total cost is selected: x = [510, 290, 720, 390, 90], f1 = 35, f2 = 8500.

[0090] 3D visualization engine 1032

[0091] Multi-source data fusion layer 1032a: UAV LiDAR point cloud and BIM model are registered by improving ICP algorithm, confidence weight ω i Determined by the point cloud density. The registration error threshold is set to 0.1m and convergence is achieved after 50 iterations.

[0092] Real-time rendering layer 1032b: uses WebGL 2.0 to draw the 3D model. The LOD mechanism switches the level of detail according to the viewing distance: when the viewing distance is <100m, a fine model (100,000 polygons) is displayed, and when the viewing distance is ≥500m, it switches to a simple model (10,000 polygons).

[0093] 4. Intelligent interactive terminal 104

[0094] 1) Connecting to the central server 103: The intelligent interactive terminal is connected to the central server via a wired or wireless method to realize data interaction and instruction sending.

[0095] 2) If Figure 5 As shown, the functional modules include:

[0096] Dialect Adaptation Layer 1041: Adopts an adversarial transfer learning framework. The pre-trained model is based on a 1000-hour Mandarin speech corpus, and dialect data (200 hours) is added during fine-tuning. The loss function weight λ = 0.5, balancing the CTC loss and the domain difference loss.

[0097] Contextual Understanding Module 1042: The compressed BERT model has 30% of the original model parameters (the number of layers is reduced from 12 to 6, and the hidden layer dimension is reduced from 768 to 384), and retains 90% of the intent recognition accuracy through knowledge distillation.

[0098] For example, a user's voice command "Open irrigation valve No. 3" is converted into standard text by the dialect adaptation layer 1041. After parsing by the BERT model, the central server is triggered to perform the corresponding operation, with a response time of <500ms.

[0099] 5. Hierarchical data bus 105

[0100] Physical layer: OFDM modulation, number of subcarriers N = 64, cyclic prefix CP = 16, symbol duration 4 μs, bandwidth 20 MHz.

[0101] TDMA time slot allocation: When the number of access nodes M = 50, the time slot width Δt = 10ms × (1 + 0.1 × log250) = 12.3ms, and each frame contains 8 time slots. TDMA time slot allocation ensures that each node sends data within the designated time slot, avoiding data conflicts and improving data transmission efficiency and reliability.

[0102] 6. Anomaly Detection Module 106

[0103] LSTM-AE structure: input dimension is 12 (such as soil moisture, temperature, etc.), encoder hidden layer dimension is 4, and decoder hidden layer dimension is 12.

[0104] Abnormality determination: Calculate the historical data abnormality score Q3 = 15.2, IQR = 5.7, and threshold θ = 15.2 + 1.5 × 5.7 = 23.75. When A(t) > 23.75, an alarm is triggered.

[0105] For example: at a certain moment, the sensor data x(t) = [25.3, 0.45, ...], the reconstructed output Hidden layer statistics σ(t)=1.2(μ σ =0.9), then Judged as normal.

[0106] The workflow of this embodiment is:

[0107] Step 1: Multimodal data collection and transmission

[0108] Step 1.1 Sensor network initialization

[0109] Deployment configuration: The farmland area is divided according to the Voronoi diagram. Soil sensors (to monitor pH and humidity), weather stations (to monitor temperature, humidity, and wind speed), and video surveillance equipment are deployed according to the density formula ρ = K·ln(S·δ+1).

[0110] UAV path planning: using improved ant colony algorithm to generate inspection paths and pheromone update rules The drone is equipped with a multispectral camera to periodically scan the farmland (once every 2 hours) and transmit crop growth data in real time.

[0111] Step 1.2: Real-time processing at the edge

[0112] Data preprocessing:

[0113] Noise reduction: The sensor's raw signal y(t) is input into the FPGA, and a hybrid noise reduction algorithm is executed:

[0114]

[0115] Dynamic weight adjustment: If the signal-to-noise ratio is less than 15dB, α = 0.8 (focusing on wavelet threshold denoising); when the signal-to-noise ratio is ≥ 15dB, α = 0.4 (focusing on Kalman filtering).

[0116] Feature extraction: Time series features are extracted through the 1D-CNN model, and the network outputs a 128-dimensional feature vector, such as farmland humidity trends and temperature fluctuation cycles.

[0117] Data verification and caching: CRC-32C verifies data integrity, and LRU-K strategy manages cache (retains data with a recent access frequency ≥ 2).

[0118] Step 2: Central server intelligent decision-making

[0119] Step 2.1: Hierarchical data transmission

[0120] Time trigger mechanism: data bus transmission period T = min{2 n ·T0|n∈N}, reference period T0=1s.

[0121] TDMA time slot allocation: When the number of access nodes M = 50, the time slot width Δt = 10ms × (1 + 0.1 × log250) = 12.3ms, ensuring conflict-free transmission in high-concurrency scenarios.

[0122] Step 2.2: Multi-objective resource optimization

[0123] NSGA-II algorithm execution:

[0124] Initialization: Generate 200 initial solutions, decision variable x j Indicates the amount of hydropower resources allocated.

[0125] Crossover mutation: adaptive crossover probability p c = 0.9-0.5·(g / 100), when it evolves to the 50th generation, p c =0.65.

[0126] Local search: Perform simulated annealing optimization on the non-dominated solution, with the temperature decay formula T(k) = 100·0.95 k , after 10 iterations, the temperature dropped to 59.9.

[0127] Pareto solution set output: Generates a set of resource allocation plans for administrators to choose a plan that balances cost and error.

[0128] Step 2.3: 3D visualization

[0129] Point cloud registration: Improved ICP algorithm to fuse UAV LiDAR point cloud and BIM model, objective function min∑ω i ·||R·p i +t―q i || 2 , confidence weight ω i Dynamically assigned value according to point cloud density.

[0130] Real-time rendering: The WebGL engine switches the LOD model according to the user's viewing distance:

[0131] Viewing distance <100m: Displays a high-precision model of 100,000 polygons (including crop leaf textures);

[0132] View distance ≥ 500m: Switch to a simplified model with 10,000 polygons (only retaining the land outline).

[0133] Step 3: Anomaly Detection and User Interaction

[0134] Step 3.1: Abnormal monitoring and alarm

[0135] LSTM-AE autoencoder: Input 12-dimensional sensor data (such as soil moisture, temperature, CO2 concentration), the hidden layer dimension is 4, and the reconstruction error is calculated as:

[0136]

[0137] Gradual alarm: When A(t)>Q3+1.5·IQR, the alarm is triggered (if the historical data Q3=18.6, Q3=18.6, IQR=6.2, the threshold θ=18.6+9.3=27.9).

[0138] Step 3.2: Multimodal Human-Computer Interaction

[0139] Voice command processing:

[0140] Dialect adaptation: The adversarial transfer learning framework maps dialect speech to the standard feature space, with the loss function L = L CTC +0.5·L domain .

[0141] Intent parsing: The compressed BERT model parses user commands (such as "show the irrigation status of plot 3") with a response delay of <300ms.

[0142] Visual feedback: The 3D engine highlights the target area and overlays a real-time data panel (such as water pump switch status and water level change curve).

[0143] Step 4: Closed-loop control and system update

[0144] Step 4.1: Actuator control

[0145] Automatic response: The central server sends instructions to the farmland irrigation system, street light controller and other equipment. When the soil moisture is lower than the threshold, the precision irrigation valve is triggered and the water volume is calculated according to the optimization result x j =85m 3 / ha allocation.

[0146] Step 4.2: Dynamic model update

[0147] Incremental Learning:

[0148] The resource allocation model is updated monthly to incorporate the latest cost parameters and demand data;

[0149] The speech recognition model integrates dialect data from various terminals through federated learning to protect user privacy.

[0150] Compared with traditional rural management systems, this embodiment uses multimodal data fusion (soil, meteorological, video and drone collaborative perception) instead of single sensor collection, increasing data coverage and real-time performance by three times. Edge computing nodes compress data processing delays from >500ms to less than 10ms, reducing bandwidth usage by 70%. The improved NSGA-II algorithm achieves multi-objective optimization of hydropower resources, reducing the error rate from 25% based on traditional manual experience to 5%, and reducing overall costs by more than 20%. The BIM-GIS-based 3D visualization engine supports centimeter-level precision modeling and voice interaction, replacing traditional 2D static charts, increasing decision-making response speed by 50 times, and promoting the transition of rural management from experience-driven to data-intelligent closed-loop.

[0151] The above only describes in detail the preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the purpose of the present invention, and various changes should be included in the scope of protection of the present invention.

Claims

1. A digital village intelligent management system based on multimodal data fusion, characterized by: include: A heterogeneous sensor network (101) is deployed in farmland, roads, and public facilities, including soil composition sensors, meteorological sensors, video acquisition equipment, and drone inspection modules. Each sensor is hierarchically networked through the LoRaWAN protocol. An edge computing node (102), the edge computing node (102) being connected to the heterogeneous sensor network (101), the edge computing node (102) having a built-in FPGA pre-processing module and configured with a hybrid noise reduction algorithm based on wavelet transform-Kalman filtering; The central server (103) is connected to the edge computing node (102) via a hierarchical data bus (105). The hierarchical data bus (105) adopts a time-triggered communication mechanism with a transmission period of T=min{2 n T0|n∈N}, where T0 is the reference clock period and T is the transmission period of the data bus; the central server (103) includes: Resource allocation optimization module (1031), which uses the improved NSGA-II algorithm to achieve multi-objective optimization of hydropower resources; 3D visualization engine (1032), building a digital twin village model based on BIM-GIS fusion modeling technology; An intelligent interactive terminal (104) is connected to a central server (103), and the intelligent interactive terminal (104) is equipped with a speech recognition module to support multimodal human-computer interaction.

2. According to the digital rural intelligent management system based on multimodal data fusion according to claim 1, the deployment method of the heterogeneous sensor network (101) comprises: Based on the Voronoi diagram region partitioning algorithm, the sensor layout density is ρ = K·ln(S·δ+1), where: K is a proportional constant used to calibrate the combined effects of terrain and crop factors on density; S is the terrain complexity coefficient, and δ is the crop growth index; The UAV inspection path is planned using the improved ant colony algorithm, and the pheromone update rule is: Where: τ ij (t) is the pheromone concentration from node i to node j in the UAV inspection path at time t; τ ij (t+1) is the updated pheromone concentration on edge (i, j) at time (t+1), that is, after completing one inspection iteration; is the pheromone increment of the optimal path on the edge (i, j) in the current iteration, Get the pheromone increment of the current optimal path.

3. According to the digital rural intelligent management system based on multimodal data fusion according to claim 1, the expression of the wavelet transform-Kalman filter hybrid noise reduction algorithm in the edge computing node (102) is: in: α is the dynamic weight coefficient, Q and R are the covariance matrices of process noise and observation noise respectively, y(t) is the input original sensor signal, represents the output signal after the hybrid noise reduction algorithm is processed. DWT(·) represents discrete wavelet transform, which is used to decompose the original signal into different frequency domains and extract the noise and effective signal components. -1 (·) represents the inverse discrete wavelet transform, which reconstructs the processed wavelet coefficients back into the time domain signal; Φ(·) represents the threshold processing function of the wavelet coefficients; KF(y(t)|Q,R) is the Kalman filter algorithm, the input is the original signal y(t), and the output is the filtered estimate.

4. According to the digital rural intelligent management system based on multimodal data fusion according to claim 1, the FPGA pre-processing module of the edge computing node (102) includes: A data check unit (1021) adopts a CRC-32C cyclic redundancy check code; The feature extraction unit (1022) is equipped with a temporal feature extractor based on 1D-CNN, and the network structure is [Conv(64,k=5)―MaxPool(2)―Conv(128,k=3)―GlobalAvgPool]; Among them: Conv(64,k=5) represents the first one-dimensional convolution layer, which is used to extract the local features of the time series signal collected by the sensor; the number of output channels is 64, which means that 64 different time series features are extracted; k=5 means that the convolution kernel size is 5, that is, each convolution operation considers the signals of the current moment and the previous 4 moments, capturing the local time series dependency of 5 time steps; MaxPool(2) represents the maximum pooling layer with a pooling window size of 2, which is used to reduce the dimension and retain the maximum eigenvalue of the local area; Conv(128,k=3) represents the second one-dimensional convolution layer, with an output channel number of 128 and a convolution kernel size of 3; GlobalAvgPool is a global average pooling layer, which takes the average value of the entire time dimension and compresses the multi-dimensional feature map into a feature vector of fixed length; The cache management unit (1023) implements the LRU-K replacement strategy.

5. According to the digital village intelligent management system based on multimodal data fusion according to claim 1, the objective function of the resource allocation optimization module (1031) is: in: x is the resource allocation decision variable vector, F(x) is the objective function vector of multi-objective optimization, which contains two sub-goals f1(x) and f2(x), f1(x) is the weighted absolute error sum, w i is the weight of the i-th demand indicator, a i is the actual assigned value of the i-th indicator, is the target demand value of the i-th indicator; f2(x) is the total cost of resource allocation, c j is the cost coefficient of the j-th type of resource, x j is the allocation amount of the jth type of resources.

6. According to the digital village intelligent management system based on multimodal data fusion according to claim 1, the workflow of the resource allocation optimization module (1031) includes: Establish a Pareto frontier search mechanism and use the ε-dominance strategy to maintain the diversity of the solution set; Design the adaptive crossover probability p c =0.9-0.5·(g / G), where g is the current generation and G is the total evolution generation; Introduce a local search operator and perform simulated annealing optimization on the non-dominated solution: T(k) = t0·0.95 k , where: T(k) is the temperature value at the kth iteration, which measures the probability of accepting an inferior solution during the search process; t0 is the initial temperature, which determines the exploration intensity in the initial stage of the algorithm; k is the number of annealing iterations (k = 0, 1, 2, ...), which represents the current step in the local search process.

7. The digital village intelligent management system based on multimodal data fusion according to claim 1, wherein the three-dimensional visualization engine (1032) comprises: The multi-source data fusion layer (1032a) uses the improved ICP algorithm to achieve point cloud registration, and the objective function is: min∑ω i ·||R·p i +t―q i || 2 Where: i is the confidence weight; p i Represents the i-th 3D coordinate point of the registered point cloud data, including the farmland point cloud scanned by UAV LiDAR and the road point cloud collected by vehicle-mounted LiDAR; q i Indicates the point cloud data of the registration benchmark and p i The corresponding nearest neighbor points include high-precision BIM model point clouds and GIS georeferenced point clouds; R represents a 3×3 orthogonal rotation matrix, which describes the rotational posture of the source point cloud relative to the target point cloud and contains 3 degrees of freedom; t represents a 3×1 translation vector, which describes the translation offset of the source point cloud relative to the target point cloud; Real-time rendering layer (1032b), built on WebGL 2.0, supports LOD multi-level of detail model switching.

8. According to the digital village intelligent management system based on multimodal data fusion according to claim 1, the voice recognition module of the intelligent interactive terminal (104) includes: The dialect adaptation layer (1041) adopts the adversarial transfer learning framework, and the loss function is: L=L CTC +λ·L domain Where: L is the total loss, which comprehensively measures the overall error of the speech recognition model in the speech-to-text alignment task and the domain transfer adaptation task, and guides the model to optimize the two capabilities simultaneously through weighted summation; L CTC L is a connectionist temporal classification loss, which is used in scenarios where the length of the speech feature sequence is inconsistent with the length of the text sequence; domain is the domain loss, which is used to measure the distribution difference between standard Mandarin speech data and dialect speech data, and forces the model to ignore domain-specific features through adversarial training; λ is the balance L CTC and L domain The weight of importance is λ∈[0.3,0.7]; The context understanding module (1042) integrates the BERT-Base pre-trained model and compresses the model size to 30% of the original through knowledge distillation.

9. The digital rural intelligent management system based on multimodal data fusion according to claim 1, wherein the communication protocol of the hierarchical data bus (105) includes: The physical layer uses OFDM modulation, with the number of subcarriers N = 64 and the cyclic prefix length CP = 16; The data link layer implements TDMA time slot allocation, with a time slot width of Δt = 10ms·(1+0.1·log2M), where M is the number of access nodes.

10. The digital village intelligent management system based on multimodal data fusion according to claim 1 further comprises an anomaly detection module (106), wherein the anomaly detection module (106) is connected to the central server (103), and the implementation method of the anomaly detection module (106) is as follows: Construct an LSTM-AE autoencoder with the hidden layer dimension set to 1 / 3 of the input dimension; Defining anomaly scores in: x(t) is the input data vector at the current time t; is the output data vector reconstructed by the autoencoder; ||·||2 is the L2 norm, which measures the point-by-point difference between the input data and the reconstructed data; σ(y) is the statistic of the hidden layer activation value at the current moment; μ σ is the long-term mean of the hidden layer statistics; γ is the weight parameter, γ = 0.7; When A(t)>θ, a graded alarm is triggered, where θ=Q3+1.5·IQR, where Q3 is the third quartile, representing the 75% position value of the abnormal score data set; and IQR is the interquartile range, reflecting the distribution range of the middle 50% of the data.

Images