A large-scale field low-altitude body-mounted intelligent multi-modal data collaborative generation method

Abstract

The present application relates to the field of low-altitude body intelligence, and more particularly to a large-scale field low-altitude body intelligent multi-modal data collaborative generation method; by constructing a field scene mechanism prior base to generate a scene environment prior state vector, and introducing a cross-modal feature alignment mechanism to take the state vector as a geometric and physical constraint, combining a collaborative generation architecture combining an autoregressive generation model and a diffusion generation model, effectively solving the technical problems of insufficient physical reality, poor multi-modal spatio-temporal consistency and weak task adaptability in the prior art, by establishing a field scene mechanism prior base including an altitude gradient mechanism model, a terrain illumination mechanism model, a weather mechanism model, a crop growth mechanism model and a pest and disease evolution mechanism model, systematic and unified modeling of various scene characteristics in a complex plateau mountain environment is realized, so that the generated data can truly reflect the physical and biological laws of the field scene.

Description

Technical Field

[0001] This invention relates to the field of low-altitude embodied intelligence technology, and in particular to a method for collaborative generation of large-scale low-altitude embodied intelligence multimodal data in the field. Background Technology

[0002] With the development of low-altitude economy, embodied intelligence, and smart agriculture technologies, the application of drones and related low-altitude intelligent systems in mountain agriculture, specialty economic crop cultivation, and field agricultural monitoring is constantly deepening. For Yunnan's plateau specialty agriculture such as coffee and tea, tasks such as pest and disease identification and control, park inspection, growth monitoring, yield prediction, and operation assistance place higher demands on the environmental perception, spatiotemporal understanding, decision-making reasoning, and task execution capabilities of low-altitude embodied intelligent agents.

[0003] The training, validation, and evaluation of low-altitude embodied intelligent systems rely on large-scale, high-quality, realistic, and task-adaptable multimodal data. However, compared to agricultural scenarios in plains farmland, regular orchards, or low-altitude single-climate zones, the scenarios of wild crops such as coffee and tea in Yunnan are more complex and heterogeneous.

[0004] On the one hand, the terrain of the Yunnan plateau and mountainous areas is highly undulating, with alternating slopes, valleys, terraces, forest edges, and shady slopes, leading to more complex scale variations, occlusion relationships, background textures, and perspective geometry in low-altitude observations. On the other hand, strong altitude gradients and significant local climate changes, along with cloud and fog humidity, dry-wet season transitions, strong light and backlighting, and different slope aspect microenvironments, result in significant temporal and spatial fluctuations in image appearance, multispectral response, and thermal infrared distribution. Furthermore, plateau-specific crops such as coffee and tea exhibit significant phenotypic differences at different altitudes, growth stages, and disease and pest infection stages, making it more difficult to obtain samples of mild diseases, early-stage diseases, complex occlusion diseases, and scarce yields. Existing data generation processes are often geared towards single images or single-task scenarios, lacking unified constraints on video, point clouds, environmental parameters, and task semantics, making it difficult to guarantee cross-modal consistency and label credibility.

[0005] Therefore, existing technologies have the following main shortcomings: First, insufficient physical realism. There is a lack of unified modeling of environmental mechanisms such as altitude gradient, slope aspect differences, light migration, cloud and humidity, dry and wet season variations, and crop growth and pest and disease evolution patterns. This results in generated data that, while realistic, does not truly reflect the actual field scene. Second, insufficient multimodal spatiotemporal consistency. There is a lack of strict geometric and semantic alignment between different modalities of data, such as images, videos, point clouds, environmental parameters, and task semantics. This easily leads to problems such as blurred edges, cross-viewpoint drift, 3D mismatch, and label misalignment. Third, insufficient task adaptability. The generation process is not fully geared towards specific downstream tasks such as pest and disease identification and control, inspection, and yield prediction. It is difficult to generate truly high-value, difficult-to-use samples that genuinely serve these downstream tasks, thus affecting the model's practical application performance. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method for collaborative generation of large-scale, low-altitude, embodied intelligent multimodal data in the field, which solves the technical problems of insufficient physical realism, insufficient spatiotemporal consistency of multimodal data, and insufficient task adaptability in existing technologies.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: a method for collaborative generation of large-scale low-altitude embodied intelligent multimodal data in the field, comprising the following steps: S1. Obtain basic data of the field crop scene, the basic data including at least image data, video data, three-dimensional structural data, environmental parameter data and task semantic data; S2. Based on the aforementioned basic data, construct a priori framework for field scene mechanisms, and uniformly model at least one scene feature in altitude gradient, slope aspect difference, shadow occlusion, weather conditions, cloud and fog humidity, dry and wet season transition, crop growth stage, and pest and disease evolution to generate a priori state vector of the scene environment. S3. Establish a cross-modal feature alignment mechanism to map each basic data to a unified feature space, and introduce the scene environment prior state vector as geometric and physical constraints to obtain spatiotemporally aligned multimodal representation results. S4. Based on the multimodal representation results and preset task conditions, and in conjunction with the prior state vector of the scene environment, a collaborative generation architecture combining an autoregressive generation model and a diffusion generation model is adopted to generate sample data for the corresponding modality.

[0008] Preferably, in step S2, the prior art of the field scene mechanism includes at least: An altitude gradient mechanism model used to describe the relationship between temperature and humidity, crop phenotype and imaging quality changes under different altitude conditions; A terrain lighting mechanism model used to characterize the effects of slope aspect differences, shadow occlusion, valley reflection, and backlighting conditions; Weather mechanism models used to characterize different weather and environmental transition states; A crop growth mechanism model used to characterize the appearance and structural features of various field crops at different growth stages; A pest evolution mechanism model used to describe the apparent changes, spread trends, and thermal distribution characteristics of different pest types under different environments and growth stages.

[0009] Preferably, the specific steps for constructing the priori framework of the field scene mechanism are as follows: The altitude of the collection area is stratified by a preset altitude interval. In each altitude layer, at least one parameter including temperature, humidity, solar irradiance, leaf color index and image signal-to-noise ratio is statistically analyzed. The altitude gradient response coefficient is calculated by a regression model to construct an altitude gradient mechanism model. Based on the digital elevation model, the slope and aspect data of the collection area, and the solar position parameters at the time of collection, pixel-level light intensity maps and occlusion masks are obtained by using topographic radiation transfer calculation and shadow projection calculation to construct a topographic lighting mechanism model. Multiple weather samples from the collection area are clustered, and weather state parameters are obtained by mapping the relationship between at least one parameter, including visible light contrast, fog scattering coefficient and thermal infrared temperature difference, and ambient temperature and humidity, so as to construct a weather mechanism model. We will statistically analyze at least one parameter from crop texture, canopy structure, fruit density, lesion color, and thermal characteristics at different growth stages of wild crops, and establish a time-series transition table of growth stages and pest and disease evolution to construct crop growth mechanism models and pest and disease evolution mechanism models.

[0010] Preferably, in step S3, the specific steps of the cross-modal feature alignment mechanism are as follows: The frame sequences in image and video data are normalized in size, corrected for distortion, and divided into grids, and the corresponding two-dimensional visual tokens are extracted. Normal constraint encoding is performed on the depth map in the 3D structural data, and voxelization or neighborhood sampling is performed on the point cloud data in the 3D structural data before encoding, so as to obtain the corresponding 3D structural tokens respectively. After standardizing the environmental parameter data, it is input into the multilayer perceptron and a sinusoidal position code is superimposed to obtain the state token; Semantic tokens are obtained from the task's semantic data using a text encoder; Different tokens are mapped to a unified dimension, and timestamp encoding, viewpoint pose encoding, and modality type encoding are introduced. During the mapping and encoding process, the prior state vector of the scene environment is used as a feature consistency constraint. Multimodal representation results are obtained in the shared embedding space through cross-modal contrastive learning, cross-attention fusion, and feature consistency constraints.

[0011] Preferably, during the sample data generation process, a multi-agent collaborative mechanism consisting of multiple agents is constructed to collaboratively execute task planning, environment modeling, data generation, annotation generation, quality assessment, and resource scheduling, thereby controlling the sample data generation process. The multiple agents include at least: The scene understanding agent is used to parse the scene attributes and hard sample distribution of the basic data of the field crop scene, and output the scene analysis results containing at least scene labels, terrain categories and hard sample distribution; The task planning agent is used to understand the scenario analysis results and target tasks output by the agent based on the scenario, and to generate data requirements and generation constraints. An environment modeling agent is used to invoke the prior framework of the field scene mechanism, generate and broadcast the prior state vector of the scene environment to other agents; The data generation agent is used to generate sample data for the corresponding modality based on the data requirements and generation constraints output by the task planning agent. The annotation generation agent is used to synchronously generate annotations corresponding to the sample data; A quality assessment agent is used to score the quality of the generated sample data in multiple dimensions. The resource scheduling agent is used to dynamically adjust model parameters, modal weights, sampling strategies, and task priorities based on multidimensional scoring results and computing load.

[0012] Preferably, during the sample screening stage, the quality assessment agent scores the generated samples from at least six dimensions, including physical authenticity, cross-modal alignment, spatiotemporal consistency, label credibility, sample diversity, and task adaptability. When the score is lower than a preset threshold, the resource scheduling agent triggers a feedback correction operation, which includes: model parameter correction, modality weight adjustment, sampling strategy update, and task priority reordering.

[0013] Preferably, during the model training phase, the resource scheduling agent is also used to increase the generation ratio of the corresponding samples and retrain when it is detected that the coverage of specific types of difficult samples is insufficient or the verification index drops; the specific types of difficult samples include at least one of the following samples: high-altitude area samples, slope shading samples, rain and fog samples, strong backlight samples, mild disease samples, early disease samples, or scarce yield range samples.

[0014] Preferably, the collaborative generation architecture combining the autoregressive generation model and the diffusion generation model specifically includes: The conditional coding submodule is used to receive multimodal representation results, data requirements and generation constraints, task conditions and scenario environment prior state vectors; The spatiotemporal latent variable planning submodule adopts an autoregressive Transformer, based on multimodal representation results and prior state vectors of the scene environment, to predict a sequence of autoregressive latent variables including scene layout tokens, camera trajectory tokens, target state tokens and time tokens, in order to model perspective changes, temporal changes and semantic evolution. The diffusion decoding submodule uses a two-dimensional or three-dimensional U-Net diffusion network to progressively denoise under the joint conditions of the scene environment prior state vector, the pose vector extracted from the basic data, and the autoregressive latent variable sequence. The consistency constraint submodule is used to calculate cross-frame optical flow consistency loss, cross-modal alignment loss, geometric reprojection loss, and label supervision loss, and then feeds the results back to each submodule to complete parameter updates.

[0015] Preferably, when video samples need to be generated, the spatiotemporal latent variable planning submodule first outputs the keyframe latent variables and flight trajectory sequence, and then the diffusion decoding submodule completes the intermediate frames. When it is necessary to generate a depth map or point cloud sample, the diffusion decoding submodule synchronously outputs the depth map and performs back projection or multi-view fusion based on the camera's internal and external parameters to obtain the point cloud structure. When it is necessary to generate thermal infrared or multispectral modes, the diffusion decoding submodule outputs the generation data of the corresponding mode based on the shared autoregressive latent variable sequence, and ensures cross-mode alignment through pose and spatial constraints at the same time.

[0016] A large-scale, low-altitude, embodied intelligent multimodal data collaborative generation system for the field includes a processor and a memory. The memory is used to store a computer program, which, when executed by the processor, implements the data collaborative generation method.

[0017] By employing the above technical solution, the present invention provides a method for large-scale collaborative generation of low-altitude embodied intelligent multimodal data in the field, which has at least the following beneficial effects: 1. This invention generates a prior state vector of the scene environment by constructing a prior base of field scene mechanism, and introduces a cross-modal feature alignment mechanism to use the state vector as geometric and physical constraints. Combined with a collaborative generation architecture that combines autoregressive generation model and diffusion generation model, it effectively solves the technical problems of insufficient physical realism, poor spatiotemporal consistency of multimodal and weak task adaptability in the prior art, and significantly improves the environmental realism and cross-modal matching of the generated data.

[0018] 2. This invention establishes a priori framework for field scene mechanisms, including models of altitude gradient, topographic illumination, weather, crop growth, and pest evolution. This enables systematic and unified modeling of various scene characteristics in complex plateau and mountainous environments, allowing the generated data to truly reflect the physical and biological laws of the field scene and avoiding the generation defects of "similar but incorrect" in traditional methods.

[0019] 3. This invention adopts a collaborative generation architecture that combines an autoregressive generation model and a diffusion generation model. The autoregressive generation model is used to model changes in perspective, temporal changes, and semantic evolution of the task, while the diffusion generation model is used to recover detailed textures, thermal distribution boundaries, and local spatial structures. This achieves multimodal data generation that balances global semantic consistency and local physical realism, significantly improving the adaptability of the generated samples for downstream tasks such as pest and disease identification and control, inspection, and yield prediction. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a diagram illustrating the overall technical framework of the present invention; Figure 2 This is a flowchart of the cross-modal alignment and collaborative generation architecture of the present invention; Figure 3 This is a schematic diagram of the multi-agent collaboration and closed-loop feedback mechanism of the present invention. Detailed Implementation

[0021] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This will allow for a full understanding and implementation of how the present application uses technical means to solve technical problems and achieve technical effects.

[0022] To address the technical problems of insufficient environmental mechanism modeling, inadequate multimodal spatiotemporal consistency, and insufficient task-oriented capabilities in existing technologies, this embodiment provides a large-scale, low-altitude, embodied intelligent multimodal data collaborative generation method. Targeting the field crop scenario in the Yunnan plateau and mountainous region, this method improves the realism, consistency, and task adaptability of generated data in complex environments by integrating prior environmental mechanism knowledge, cross-modal feature alignment, and task-driven approaches. Figure 1 As shown, the method includes the following steps: Step S1: Obtain basic data.

[0023] In this embodiment, firstly, basic data of the target area is collected using an unmanned aerial vehicle (UAV) platform, ground sensors, and a historical database. This basic data includes at least image data, video data, 3D structural data, environmental parameter data, and task semantic data. Image data is typically low-altitude RGB images; video data is typically low-altitude video sequences; 3D structural data typically includes depth maps or point cloud data; environmental parameter data typically includes terrain elevation data, slope and aspect information, GNSS / IMU pose data, and environmental meteorological data; task semantic data is task description text, such as "Please identify whether rust exists in the coffee plantation and assess its severity," which can be input by the user through an interactive interface. Then, the collected data undergoes time synchronization, spatial registration, coordinate unification, noise suppression, viewpoint correction, and missing data completion to obtain standardized multi-source data as the basic data for subsequent input.

[0024] Step S2: Construct a priori framework for the mechanism of the field scenario.

[0025] Based on fundamental data, a priori framework for field scene mechanisms is constructed. This framework uniformly models at least one scene feature from the following categories: elevation gradient, slope aspect differences, shading, weather conditions, cloud / fog humidity, dry / wet season transitions, crop growth stages, and pest / disease evolution. This generates a priori state vector for the scene environment. A preferred implementation is to use several of these scene features simultaneously. Specifically, the priori framework for field scene mechanisms constructed in this embodiment includes five types, as follows: 1. An altitude gradient mechanism model is used to describe the relationship between temperature and humidity, crop phenotype, and imaging quality under different altitude conditions.

[0026] 2. A terrain lighting mechanism model is used to characterize the effects of slope aspect differences, shadow occlusion, valley reflection, and backlighting conditions.

[0027] 3. Weather mechanism models are used to characterize different weather and environmental transition states, such as sunny days, cloudy days, rainy days, foggy days, high humidity environments, and dry and wet season transition states.

[0028] 4. Crop growth mechanism model, used to characterize the appearance and structural features of wild crops such as coffee and tea at different growth stages, including budding, leaf unfolding, flowering, fruiting and ripening.

[0029] 5. Pest and disease evolution mechanism model, used to describe the apparent changes, spread trends and thermal distribution characteristics of different types of pests and diseases under different environments and growth stages.

[0030] The aforementioned prior frameworks of various field scenarios collectively constitute the prior state vector of the scene environment. For example, the prior state vector of the scene environment is a one-dimensional floating-point vector with 256 dimensions. The first 64 dimensions represent the elevation gradient features, the 65th to 128th dimensions represent the terrain and illumination features, the 129th to 192nd dimensions represent the weather features, the 193rd to 224th dimensions represent the crop growth stage features, and the 225th to 256th dimensions represent the pest and disease evolution features. The values ​​of each dimension can be normalized and mapped to the [0,1] interval. Then, they are merged into the final prior state vector of the scene environment through a splicing operation, which is used to constrain the subsequent cross-modal coding, sample generation, and label generation processes. When constructing the prior art for each field scene mechanism, firstly, the altitude of the data collection area is stratified at preset altitude intervals. Within each altitude layer, at least one parameter from temperature, humidity, solar irradiance, leaf color index, and image signal-to-noise ratio is statistically analyzed. Altitude gradient response coefficients are then calculated using hierarchical linear regression or multinomial regression models to construct the altitude gradient mechanism model. Next, based on the digital elevation model, slope and aspect data of the data collection area, and solar position parameters at the time of data collection, pixel-level light intensity maps and occlusion masks are obtained using terrain radiative transfer calculations and shadow projection calculations to construct the terrain illumination mechanism model. Finally, the data collection... Clustering was performed on various weather samples from the region, including sunny, cloudy, rainy, foggy, and dry / wet season samples. Weather state parameters were obtained by mapping at least one parameter (visible light contrast, fog scattering coefficient, and thermal infrared temperature difference) to environmental temperature and humidity to construct a weather mechanism model. Finally, at least one parameter (crop texture, canopy structure, fruit density, lesion color, and thermal characteristics) was statistically analyzed for different growth stages of wild crops, such as budding, leaf expansion, flowering, fruiting, and maturity. Hidden Markov chains or gated cyclic units were used to establish a time series transition table for growth stages and pest and disease evolution to complete the construction of crop growth mechanism models and pest and disease evolution mechanism models.

[0031] Step S3: Establish a cross-modal feature alignment mechanism.

[0032] A cross-modal feature alignment mechanism is established to map each basic data to a unified feature space, and the prior state vector of the scene environment is introduced as a geometric and physical constraint to obtain a spatiotemporally aligned multimodal representation result. In this embodiment, the specific steps of the cross-modal feature alignment mechanism are as follows: The frame sequences in RGB image and video data are normalized, distortion corrected, and meshed, and then input into the backbone of a convolutional neural network or a visual Transformer to extract the corresponding 2D visual tokens. Next, normal constraint encoding is performed on the depth map in the 3D structural data, and voxelization or neighborhood sampling is performed on the point cloud data in the 3D structural data. Then, PointNet++ or sparse 3D convolutional encoding is used to obtain the corresponding 3D structural tokens. GNSS / IMU pose parameters and environmental sensor data in the environmental parameter data are standardized and input into a multilayer perceptron, and sinusoidal position encoding is superimposed to obtain state tokens. Semantic tokens are obtained from the task semantic data using a text encoder such as a Transformer. Next, the different tokens obtained above are mapped to a unified dimension d, and timestamp encoding, viewpoint pose encoding, and modality type encoding are introduced. During the mapping and encoding process, the prior state vector of the scene environment is used as a feature consistency constraint term to achieve geometric and physical constraints. In the shared embedding space, multimodal representation results are obtained through cross-modal contrastive learning, cross-attention fusion, and feature consistency constraints. In this embodiment, the positive sample pairs of cross-modal contrastive learning can be RGB image-depth map combinations and image-task semantic combinations of the same scene, while the negative sample pairs are modal combinations of different scenes. The contrast loss adopts the InfoNCE loss function. The cross-attention fusion module can adopt a multi-head attention mechanism. For the image feature sequence Q and the semantic feature sequence K / V, fused features are obtained through attention computation. The fused feature dimension remains consistent with the original feature dimension (256 dimensions). Feature enhancement is performed through residual connections and layer normalization. In the cross-modal alignment process, the scene environment prior state vector is used as a geometric and physical consistency constraint term. The geometric consistency constraint can utilize the terrain illumination features and elevation gradient features in the scene environment prior state vector to calculate the geometric projection deviation of each modality feature in three-dimensional space. Specifically, for each pair of cross-modal features (such as two-dimensional visual tokens and three-dimensional structural tokens), their reprojection error in the same three-dimensional coordinate system is calculated and minimized. The physical consistency constraint can utilize the weather state parameters and crop growth stage parameters in the scene environment prior state vector to constrain the consistency of the physical properties of each modality feature. For example, for the pixel brightness in the RGB image and the temperature value in the thermal infrared image, the physical property deviation is calculated based on the light-temperature mapping relationship in the weather mechanism model. Semantic consistency constraints can leverage the evolutionary features of pests and diseases and the crop growth features in the prior state vector of the scene environment to constrain the semantic labeling consistency of features across modalities. For example, for lesion regions in an RGB image and their corresponding regions in a depth map, the semantic labeling deviation is calculated based on the shape-depth relationship of lesions in the pest and disease evolution mechanism model. The weighted sum of the above three constraints forms the total loss function, which serves as the constraint for cross-modal alignment.The data is then input into a collaborative generation architecture that combines the autoregressive generation model and the diffusion generation model in step S4. This ultimately ensures strict matching of different modalities in geometry, semantics, and time, solving the problem of poor spatiotemporal consistency of multimodal data in existing technologies.

[0033] S4. Based on the multimodal representation results and preset task conditions, and in conjunction with the prior state vector of the scene environment, a collaborative generation architecture combining an autoregressive generation model and a diffusion generation model is adopted to generate sample data for the corresponding modality. The autoregressive generation model is used to model perspective changes, temporal changes, and task semantic evolution, while the diffusion generation model is used to recover detailed textures, heat distribution boundaries, and local spatial structures. This outputs image, video, depth, or point cloud data of the corresponding modality that matches the specific task. In this embodiment, the collaborative generation architecture combining the autoregressive generation model and the diffusion generation model includes at least: a conditional encoding submodule, a spatiotemporal latent variable planning submodule, a diffusion decoding submodule, and a consistency constraint submodule. Specifically, as follows... Figure 2 As shown, the conditional encoding submodule receives multimodal representation results, data requirements and generation constraints, task conditions, and prior state vectors of the scene environment. The spatiotemporal latent variable planning submodule uses an autoregressive Transformer to predict an autoregressive latent variable sequence containing scene layout tokens, camera trajectory tokens, target state tokens, and time tokens based on the multimodal representation results and prior state vectors of the scene environment. The latent variable sequence is predicted in the order of "scene layout tokens, camera trajectory tokens, target state tokens, and time tokens," meaning the autoregressive latent variable sequence contains four consecutive tokens. Each token can be a 128-dimensional floating-point vector, and the sequence order is fixed as follows: the first token is the scene layout token, describing the global layout information of the scene (such as terrain contours and crop distribution); the second token is the camera trajectory token, describing the pose changes of the UAV over time; the third token is the target state token, describing the position, size, and category information of the target object; and the fourth token is the time token, encoding the time step information of the current generation step. The above four tokens are generated sequentially to form a complete autoregressive latent variable sequence, enabling global planning of the generation process to model changes in perspective, temporal changes, and semantic evolution. The diffusion decoding submodule employs a 2D or 3D U-Net diffusion network to progressively denoise under the joint conditions of the scene environment prior state vector, the pose vector extracted from the basic data, and the autoregressive latent variable sequence, restoring detailed textures, thermal distribution boundaries, and local spatial structure. The consistency constraint submodule calculates cross-frame optical flow consistency loss, cross-modal alignment loss, geometric reprojection loss, and label supervision loss, and feeds the results back to each submodule to update parameters. Through the constraints of these various loss functions, the consistency of the generated sample data across time series, multiple modalities, and 3D space is ensured. In this embodiment, the cross-frame optical flow consistency loss can use an optical flow estimation algorithm (such as RAFT) to calculate the optical flow field between adjacent frames, and then use the endpoint error (EPE) between the predicted optical flow and the actual optical flow as the loss. The cross-modal alignment loss can use the InfoNCE contrastive learning loss, projecting feature vectors from different modalities (such as RGB images and depth maps) into the same embedding space, maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. Geometric reprojection loss is calculated by projecting the 3D point cloud onto the 2D image plane based on camera intrinsic and extrinsic parameters, and then calculating the Euclidean distance between the projected points and their corresponding image pixels. As for the label supervision loss, cross-entropy loss is used for classification tasks (such as pest and disease category recognition), while Smooth L1 loss is used for detection tasks (such as bounding box regression). The total label loss is a weighted sum of the losses from each task.

[0034] When video samples need to be generated, the spatiotemporal latent variable planning submodule first outputs keyframe latent variables and flight trajectory sequences, and then the diffusion decoding submodule completes the intermediate frames. When depth maps or point cloud samples need to be generated, the diffusion decoding submodule synchronously outputs depth maps and performs back projection or multi-view fusion based on camera intrinsic and extrinsic parameters to obtain the point cloud structure. When thermal infrared or multispectral modes need to be generated, the diffusion decoding submodule outputs the corresponding mode's generated data based on the shared autoregressive latent variable sequence, and ensures cross-modal alignment through pose and spatial constraints at the same time. Through the above collaborative generation architecture, this embodiment can generate multimodal sample data such as images, videos, depth, or point clouds that match specific tasks, while taking into account global semantic consistency and local physical realism.

[0035] Construct a multi-agent cooperation mechanism. For example... Figure 3As shown, in this embodiment, a multi-agent collaborative mechanism composed of multiple agents is further constructed to collaboratively execute task planning, environment modeling, data generation, annotation generation, quality assessment, and resource scheduling, in order to control the sample data generation process and improve the sample data generation efficiency. In this embodiment, each agent includes an input adaptation layer, a task memory unit, a policy reasoning unit, a tool execution unit, and a message interface. The input adaptation layer receives a unified spatiotemporal semantic representation, an environment state vector, and task instructions. The task memory unit caches the previous generation configuration, quality score, and list of difficult samples. The policy reasoning unit outputs action instructions, which can be based on a preset rule engine (such as If-Then rules) or a lightweight decision model (such as a random forest), according to the input data and historical information in the task memory unit, such as generating samples and adjusting sampling strategies. The tool execution unit calls corresponding algorithm modules, such as autoregressive generation models, diffusion decoding submodules, and loss calculation modules, and feeds back the execution results to the policy reasoning unit. The message interface broadcasts the state to other agents. The multiple agents include at least: 1. Scene understanding agent, used to parse the scene attributes and hard sample distribution of basic data of field crop scenes, and output scene analysis results that include at least scene labels, terrain categories and hard sample distribution.

[0036] 2. Task planning agent, used to understand the scene analysis results and target tasks output by the scene understanding agent, parse the task conditions from the task semantic data, and generate data requirements and generation constraints.

[0037] 3. An environment modeling agent is used to call upon the prior foundation of the field scene mechanism, generate and broadcast the prior state vector of the scene environment to other agents.

[0038] 4. Data generation agent, used to generate sample data for the corresponding modality based on the data requirements and generation constraints output by the task planning agent.

[0039] 5. Label generation agent, used to synchronously generate labels corresponding to sample data, including target boxes, segmentation masks, category labels, time series labels, and task labels.

[0040] 6. Quality assessment agent, used to score the quality of sample data generated by the data generation agent in multiple dimensions.

[0041] 7. Resource scheduling agent, used to dynamically allocate resources based on multidimensional scoring results, sample scarcity, and computing load.

[0042] During data processing, the scene understanding agent first reads the raw input and outputs scene labels, terrain categories, and distribution of difficult samples; the task planning agent generates a task graph and modality requirement table based on this, serving as data requirements and generation constraints; the environment modeling agent calls the prior foundation of field scene mechanisms to generate a prior state vector of the scene environment; the data generation agent calls a collaborative generation architecture combining an autoregressive generation model and a diffusion generation model to generate sample data; the annotation generation agent synchronously generates annotations based on camera parameters, point cloud reprojection, and lesion masks; the quality assessment agent outputs multidimensional scores; and the resource scheduling agent dynamically adjusts model parameters, modality weights, sampling strategies, and task priorities based on the scoring results and computing load, thus forming a collaborative closed loop.

[0043] During the sample selection phase, the quality assessment agent scores the generated sample data from at least six dimensions: physical authenticity, cross-modal alignment, spatiotemporal consistency, label credibility, sample diversity, and task suitability. When the score is lower than a preset threshold, the resource scheduling agent triggers feedback correction operations, including parameter correction, modal weight adjustment, sampling strategy update, or task priority reordering, to form a closed-loop optimization and select candidate samples or final output sample data for the next round of training.

[0044] During the model training phase, supervised pre-training is first performed on the visual encoder, text encoder, spatiotemporal latent variable planning submodule, and diffusion decoding submodule using real-collected and manually verified basic data. Reconstruction loss, cross-modal contrast loss, temporal consistency loss, and geometric reprojection loss are used as basic losses. Then, multimodal sample data generated in this embodiment is mixed with real samples to form a training set, and the spatiotemporal latent variable planning submodule and diffusion decoding submodule are jointly fine-tuned. Task alignment training is performed using downstream task losses such as pest and disease identification, prevention and control suggestions, inspection anomaly detection, and yield range prediction. During the training process, periodic verification is performed according to the division of training set, validation set, and evaluation set. When the verification index decreases or the coverage of difficult samples is insufficient, the resource scheduling agent increases the generation ratio of the corresponding samples and retrains. The difficult samples of a specific type include at least one of the following: high-altitude area samples, slope shading samples, rain and fog samples, strong backlight samples, mild disease samples, early disease samples, or scarce yield range samples. After training is completed, a multimodal dataset and corresponding annotation set are output for training, validation and evaluation of the low-altitude embodied intelligent model.

[0045] The following section uses the task of identifying and controlling pests and diseases as an example to illustrate the multi-agent collaboration mechanism. The task planning agent sets the pest and disease identification and control task, with input parameters including pest and disease category, severity, crop type, growth stage, altitude range, weather conditions, collection perspective, and whether thermal infrared or multispectral enhancement information is needed. The environment modeling agent, based on the input parameters, calls the pest and disease evolution mechanism model and the weather mechanism model to construct the environmental conditions for generating pest and disease samples. For example, it simulates blurred lesion edges, abnormal leaf temperature distribution, and local color difference changes under high humidity and fog conditions, and simulates complex samples with mild lesions that are difficult to distinguish under strong light and backlight conditions. The data generation agent generates joint samples of RGB images and video sequences based on a unified spatiotemporal semantic representation. The spatiotemporal latent variable planning submodule is used to construct the diffusion and evolution process of the disease in a continuous time series, and the diffusion decoding submodule is used to refine lesion texture, leaf vein structure, and affected boundaries. The annotation generation agent simultaneously outputs pest and disease category labels, lesion detection boxes, lesion semantic segmentation masks, severity level labels, temporal diffusion trend labels, and control suggestion labels. The quality assessment agent evaluates samples based on factors such as lesion identifiability, boundary integrity, thermal feature consistency, temporal diffusion rationality, and label credibility, and prioritizes supplementing data for samples with mild diseases, shading diseases, and early-stage diseases.

[0046] The following section uses a typical coffee or tea plantation area with significant altitude gradients, slope aspect differences, frequent cloud and fog, and strong shading as the target scenario to further illustrate the multi-agent collaboration mechanism. The environmental modeling agent constructs a priori state vector for the scene environment based on altitude ranges, slope aspect categories, dry and wet season information, sunshine duration, cloud and fog humidity, terrain shading degree, and crop growth status. The multimodal generation agent generates corresponding RGB images, multispectral images, thermal infrared images, video sequences, depth structure data, and pose information based on the priori state vector. The annotation generation agent further generates labels for pest and disease categories, anomaly inspections, fruit density, yield ranges, and environmental conditions. The quality assessment agent focuses on evaluating the imaging realism under high-altitude conditions, the consistency of slope shading and shadow areas, modal matching under rain and fog conditions, target boundary clarity under strong backlight conditions, and semantic stability between samples across time periods.

[0047] This embodiment also provides a large-scale field low-altitude embodied intelligent multimodal data collaborative generation system, including a processor and a memory. The memory is used to store computer programs, and when the computer programs are executed by the processor, they implement a large-scale field low-altitude embodied intelligent multimodal data collaborative generation method.

[0048] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0049] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Since the above embodiments are substantially similar to the method embodiments, their descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0050] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for collaborative generation of large-scale low-altitude embodied intelligent multimodal data in the field, characterized in that, The method includes the following steps: S1. Obtain basic data of the field crop scene, the basic data including at least image data, video data, three-dimensional structural data, environmental parameter data and task semantic data; S2. Based on the aforementioned basic data, construct a priori framework for field scene mechanisms, and uniformly model at least one scene feature in altitude gradient, slope aspect difference, shadow occlusion, weather conditions, cloud and fog humidity, dry and wet season transition, crop growth stage, and pest and disease evolution to generate a priori state vector of the scene environment. S3. Establish a cross-modal feature alignment mechanism to map each basic data to a unified feature space, and introduce the scene environment prior state vector as geometric and physical constraints to obtain spatiotemporally aligned multimodal representation results. S4. Based on the multimodal representation results and preset task conditions, and in conjunction with the prior state vector of the scene environment, a collaborative generation architecture combining an autoregressive generation model and a diffusion generation model is adopted to generate sample data for the corresponding modality.

2. The data collaborative generation method according to claim 1, characterized in that, In step S2, the prior art of the field scene mechanism includes at least: An altitude gradient mechanism model used to describe the relationship between temperature and humidity, crop phenotype and imaging quality changes under different altitude conditions; A terrain lighting mechanism model used to characterize the effects of slope aspect differences, shadow occlusion, valley reflection, and backlighting conditions; Weather mechanism models used to characterize different weather and environmental transition states; A crop growth mechanism model used to characterize the appearance and structural features of various field crops at different growth stages; A pest evolution mechanism model used to describe the apparent changes, spread trends, and thermal distribution characteristics of different pest types under different environments and growth stages.

3. The data collaborative generation method according to claim 2, characterized in that, The specific steps for constructing the priori framework of the field scenario mechanism are as follows: The altitude of the collection area is stratified by a preset altitude interval. In each altitude layer, at least one parameter including temperature, humidity, solar irradiance, leaf color index and image signal-to-noise ratio is statistically analyzed. The altitude gradient response coefficient is calculated by a regression model to construct an altitude gradient mechanism model. Based on the digital elevation model, the slope and aspect data of the collection area, and the solar position parameters at the time of collection, pixel-level light intensity maps and occlusion masks are obtained by using topographic radiation transfer calculation and shadow projection calculation to construct a topographic lighting mechanism model. Multiple weather samples from the collection area are clustered, and weather state parameters are obtained by mapping the relationship between at least one parameter, including visible light contrast, fog scattering coefficient and thermal infrared temperature difference, and ambient temperature and humidity, so as to construct a weather mechanism model. We will statistically analyze at least one parameter from crop texture, canopy structure, fruit density, lesion color, and thermal characteristics at different growth stages of wild crops, and establish a time-series transition table of growth stages and pest and disease evolution to construct crop growth mechanism models and pest and disease evolution mechanism models.

4. The data collaborative generation method according to claim 1, characterized in that, In step S3, the specific steps of the cross-modal feature alignment mechanism are as follows: The frame sequences in image and video data are normalized in size, corrected for distortion, and divided into grids, and the corresponding two-dimensional visual tokens are extracted. Normal constraint encoding is performed on the depth map in the 3D structural data, and voxelization or neighborhood sampling is performed on the point cloud data in the 3D structural data before encoding, so as to obtain the corresponding 3D structural tokens respectively. After standardizing the environmental parameter data, it is input into the multilayer perceptron and a sinusoidal position code is superimposed to obtain the state token; Semantic tokens are obtained from the task's semantic data using a text encoder; Different tokens are mapped to a unified dimension, and timestamp encoding, viewpoint pose encoding, and modality type encoding are introduced. During the mapping and encoding process, the prior state vector of the scene environment is used as a feature consistency constraint. Multimodal representation results are obtained in the shared embedding space through cross-modal contrastive learning, cross-attention fusion, and feature consistency constraints.

5. The data collaborative generation method according to claim 1, characterized in that, During the generation of sample data, a multi-agent collaborative mechanism consisting of multiple agents is constructed to coordinate the execution of task planning, environment modeling, data generation, annotation generation, quality assessment, and resource scheduling, thereby controlling the sample data generation process. The multiple agents include at least: The scene understanding agent is used to parse the scene attributes and hard sample distribution of the basic data of the field crop scene, and output the scene analysis results containing at least scene labels, terrain categories and hard sample distribution; The task planning agent is used to understand the scenario analysis results and target tasks output by the agent based on the scenario, and to generate data requirements and generation constraints. An environment modeling agent is used to invoke the prior framework of the field scene mechanism, generate and broadcast the prior state vector of the scene environment to other agents; The data generation agent is used to generate sample data for the corresponding modality based on the data requirements and generation constraints output by the task planning agent. The annotation generation agent is used to synchronously generate annotations corresponding to the sample data; A quality assessment agent is used to score the quality of the generated sample data in multiple dimensions. The resource scheduling agent is used to dynamically adjust model parameters, modal weights, sampling strategies, and task priorities based on multidimensional scoring results and computing load.

6. The data collaborative generation method according to claim 5, characterized in that, During the sample screening phase, the quality assessment agent scores the generated samples from at least six dimensions, including physical authenticity, cross-modal alignment, spatiotemporal consistency, label credibility, sample diversity, and task suitability. When the score is lower than a preset threshold, the resource scheduling agent triggers a feedback correction operation, which includes: model parameter correction, modality weight adjustment, sampling strategy update, and task priority reordering.

7. The data collaborative generation method according to claim 5, characterized in that, During the model training phase, the resource scheduling agent is also used to increase the generation ratio of the corresponding samples and retrain when it detects insufficient coverage of specific types of difficult samples or a decline in validation metrics; the specific types of difficult samples include at least one of the following: high-altitude area samples, slope shading samples, rain and fog samples, strong backlight samples, mild disease samples, early disease samples, or samples from scarce production ranges.

8. The data collaborative generation method according to claim 1, characterized in that, The collaborative generative architecture combining the autoregressive generative model and the diffusion generative model specifically includes: The conditional coding submodule is used to receive multimodal representation results, data requirements and generation constraints, task conditions and scenario environment prior state vectors; The spatiotemporal latent variable planning submodule adopts an autoregressive Transformer, based on multimodal representation results and prior state vectors of the scene environment, to predict a sequence of autoregressive latent variables including scene layout tokens, camera trajectory tokens, target state tokens and time tokens, in order to model perspective changes, temporal changes and semantic evolution. The diffusion decoding submodule uses a two-dimensional or three-dimensional U-Net diffusion network to progressively denoise under the joint conditions of the scene environment prior state vector, the pose vector extracted from the basic data, and the autoregressive latent variable sequence. The consistency constraint submodule is used to calculate cross-frame optical flow consistency loss, cross-modal alignment loss, geometric reprojection loss, and label supervision loss, and then feeds the results back to each submodule to complete parameter updates.

9. The data collaborative generation method according to claim 8, characterized in that: When video samples need to be generated, the spatiotemporal latent variable planning submodule first outputs the keyframe latent variables and flight trajectory sequence, and then the diffusion decoding submodule completes the intermediate frames. When it is necessary to generate a depth map or point cloud sample, the diffusion decoding submodule synchronously outputs the depth map and performs back projection or multi-view fusion based on the camera's internal and external parameters to obtain the point cloud structure. When it is necessary to generate thermal infrared or multispectral modes, the diffusion decoding submodule outputs the generation data of the corresponding mode based on the shared autoregressive latent variable sequence, and ensures cross-mode alignment through pose and spatial constraints at the same time.

10. A system for implementing the data collaborative generation method according to any one of claims 1-9, characterized in that, It includes a processor and a memory, the memory being used to store a computer program that, when executed by the processor, implements the data collaborative generation method as described in any one of claims 1-9.

Images