Process analysis method and device based on multimodal data

Abstract

The present application relates to a multimodal data-based history analysis method, device, gardening robot, and storage medium. The method comprises: obtaining a historical multimodal dataset of the plant to be analyzed; invoking a plant growth analysis model; and outputting plant growth history data corresponding to the historical multimodal dataset of the plant to be analyzed through the plant growth analysis model; the plant growth history data includes growth process reconstructed text data and growth process reconstructed image data. This method enables the plant growth analysis model to reconstruct the plant's growth history, including detailed characteristics of key stages such as morphological changes and physiological development. This facilitates the observation and analysis of the specific states of the plant at different time points, not only contributing to a deeper understanding of the mechanisms of plant growth, but also providing rich basic data for subsequent prediction and anomaly detection.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a process analysis method and device based on multimodal data. Background Art

[0002] With the rapid development of artificial intelligence technology, plant-related robots have emerged, which are mostly concentrated in large-scale agricultural harvesting, picking and other operations. Although their degree of automation is already high, the application of plant-related robots in the field of horticulture is relatively limited.

[0003] Traditional gardening robots mostly follow preset parameters and rules for mechanized operations, such as watering and fertilizing according to schedule and quantity. They rely primarily on machine vision and positioning technology to ensure operational accuracy. While this model achieves basic automation, it falls short of truly intelligent gardening management. They are unable to flexibly adjust their operating strategies based on real-time environmental conditions, individual plant differences, and growth status, and lack a deep understanding of the plant growth process. Summary of the Invention

[0004] Based on this, it is necessary to provide a process analysis method and device based on multimodal data that can deeply understand the plant growth process in order to address the above technical problems.

[0005] In a first aspect, the present application provides a process analysis method based on multimodal data, comprising:

[0006] Obtain a historical multimodal dataset of the plants to be analyzed;

[0007] Calling the plant growth analysis model;

[0008] The plant multimodal dataset is input into the spatiotemporal encoder of the plant growth analysis model. The spatiotemporal encoder includes a first spatiotemporal convolutional layer, a second spatiotemporal convolutional layer, and a fully connected layer. The time series dataset of the plant to be analyzed is input into the first spatiotemporal convolutional layer to obtain the plant growth time characteristics; and the image dataset of the plant to be analyzed is input into the second spatiotemporal convolutional layer to obtain the plant growth image characteristics. The time series dataset and the image dataset are obtained by preprocessing the plant multimodal dataset;

[0009] Performing feature fusion on the plant growth time feature and the plant growth image feature through a fully connected layer to obtain a spatiotemporal feature vector;

[0010] The spatiotemporal feature vector is input into the spatiotemporal decoder of the plant growth analysis model to obtain plant growth process data; the plant growth process data includes growth process reconstructed text data and growth process reconstructed image data.

[0011] In one embodiment, the spatiotemporal feature vector is input into a spatiotemporal decoder of a plant growth analysis model to obtain plant growth history data, including:

[0012] The spatiotemporal feature vector is input into the spatiotemporal deconvolution layer of the spatiotemporal decoder. The temporal resolution and spatial resolution of the spatiotemporal feature vector are increased by the spatiotemporal deconvolution layer to obtain the plant growth process data.

[0013] In one embodiment, obtaining a historical multimodal dataset of plants to be analyzed includes:

[0014] Acquire historical multi-dimensional data of the plant to be analyzed to obtain multimodal plant data; the historical multi-dimensional data at least includes plant physiological state data, plant morphological characteristic data, and environmental impact data;

[0015] Performing smoothing and standardization on the multimodal plant data to obtain processed multimodal plant data;

[0016] Performing time stamp synchronization processing on the processed multimodal plant data to obtain synchronized multimodal plant data; aligning time stamps of the synchronized multimodal plant data at the same time point;

[0017] Associating each data in the synchronized multimodal plant data with the corresponding plant individual identifier to obtain traceable multimodal plant data;

[0018] Data integration is performed on traceable multimodal plant data to obtain a plant multimodal dataset.

[0019] In one embodiment, the process analysis method based on multimodal data further includes:

[0020] Get the number of future time steps;

[0021] The plant growth analysis model also outputs plant future growth prediction data corresponding to the future time steps of the plant to be analyzed; the plant future growth prediction data includes key growth indicator prediction text data and growth status prediction image data.

[0022] In one embodiment, the plant growth analysis model further outputs plant future growth prediction data corresponding to future time steps of the plant to be analyzed, including:

[0023] The spatiotemporal feature vectors are input into the adaptive graph neural network in the plant growth analysis model to obtain an adaptive graph structure. The spatiotemporal feature vectors are obtained by processing the historical multimodal dataset using the spatiotemporal encoder of the plant growth analysis model. The adaptive graph structure is used to reflect the intrinsic correlation between different data modalities.

[0024] The adaptive graph structure and future time steps are input into the spatiotemporal decoder in the plant growth analysis model to obtain the plant future growth prediction data.

[0025] In one embodiment, the adaptive graph neural network further includes a memory unit layer, and the process analysis method based on multimodal data further includes:

[0026] The dependency information of plant growth time characteristics is extracted and processed through the memory unit layer to obtain long-term dependency information. The plant growth time characteristics are processed by the first spatiotemporal convolutional layer in the spatiotemporal encoder to obtain the time series dataset of the plants to be analyzed. The long-term dependency information is used to represent the historical growth trends and patterns of the plants to be analyzed.

[0027] The long-term dependency information is input into the adaptive graph neural network as additional node features to obtain an adaptive graph structure.

[0028] In one embodiment, the training process of the adaptive graph neural network is as follows:

[0029] Constructing an initial undirected graph; the initial undirected graph is used to represent an undirected graph in which potential connections exist between nodes and the initial connection weight is less than an initial threshold; the nodes include plant growth parameters and the number of environmental variables;

[0030] The spatiotemporal feature vector is used as the input of the initial undirected graph, and each spatiotemporal feature vector corresponds to a node of the initial undirected graph;

[0031] Update the hidden state of the node based on the node's own features and neighbor node features in the initial undirected graph;

[0032] The edge weights between nodes are updated through the hidden states of the nodes and the global context information to obtain an adaptive graph neural network.

[0033] In one embodiment, updating the hidden state of a node based on the node's own features and neighbor node features in an initial undirected graph includes:

[0034] Memory units are embedded in the node update process, so that each node has a corresponding memory state. The memory state is used to store the node's historical information.

[0035] In each message passing iteration, the node receives information from its neighbor nodes and interacts with the node's own memory state to update the node's memory state and hidden state.

[0036] In one embodiment, the process analysis method based on multimodal data further includes:

[0037] Acquire a single-frame plant image of the plant to be analyzed;

[0038] Through the real-time anomaly detection module in the plant growth analysis model, leaf feature extraction is performed on the single-frame plant image to obtain the leaf features corresponding to the plant to be analyzed;

[0039] According to the leaf characteristics, determine whether there are abnormalities in the leaves of the plant to be analyzed;

[0040] When there is an abnormality in the leaves of the plant to be analyzed, an immediate abnormality warning signal and an immediate abnormality detail report for the plant to be analyzed are output.

[0041] In one embodiment, the process analysis method based on multimodal data further includes:

[0042] When there is no abnormality in the leaves of the plant to be analyzed, obtaining plant time series data of the current time and the plant to be analyzed;

[0043] Outputting the plant growth prediction data corresponding to the current time of the plant to be analyzed through the spatiotemporal decoder in the plant growth analysis model;

[0044] The plant current time growth prediction data and the plant time series data are input into the time series anomaly analysis module in the plant growth analysis model for comparison;

[0045] When the difference between the plant's current growth prediction data and the plant's time series data exceeds a preset threshold, it is confirmed that the plant to be analyzed has growth abnormalities, and a time series abnormality warning signal and a time series abnormality detail report for the plant to be analyzed are generated;

[0046] When the difference between the plant's current growth prediction data and the plant's time series data does not exceed a preset threshold, it is confirmed that the plant to be analyzed has no growth abnormality, and good plant growth status information is generated for the plant to be analyzed.

[0047] In one embodiment, the process analysis method based on multimodal data further includes:

[0048] Based on the external data source or the internal state of the plant growth analysis model, the key factor data affecting plant growth is estimated; the internal state of the plant growth analysis model is used to represent the state of the data related to the key factor data in the multimodal plant dataset after being processed by the plant growth analysis model;

[0049] Dynamically adjust preset thresholds based on key factor data.

[0050] In a second aspect, the present application further provides a process analysis device based on multimodal data, comprising:

[0051] Input module, used to obtain historical multimodal datasets of plants to be analyzed;

[0052] Model calling module, used to call plant growth analysis model;

[0053] A spatiotemporal convolution module is used to input a plant multimodal dataset into a spatiotemporal encoder of a plant growth analysis model. The spatiotemporal encoder includes a first spatiotemporal convolution layer, a second spatiotemporal convolution layer, and a fully connected layer. The time series dataset of the plant to be analyzed is input into the first spatiotemporal convolution layer to obtain plant growth time features, and the image dataset of the plant to be analyzed is input into the second spatiotemporal convolution layer to obtain plant growth image features. The time series dataset and the image dataset are obtained by preprocessing the plant multimodal dataset.

[0054] A feature fusion module is used to fuse plant growth time features and plant growth image features through a fully connected layer to obtain a spatiotemporal feature vector;

[0055] The output module is used to input the spatiotemporal feature vector into the spatiotemporal decoder of the plant growth analysis model to obtain plant growth history data; the plant growth history data includes growth process reconstructed text data and growth process reconstructed image data.

[0056] The multimodal data-based history analysis method and device reconstruct the plant's growth history using a plant growth analysis model based on a historical multimodal dataset of the plant being analyzed. This includes detailed features of key stages such as morphological changes and physiological development, enabling observation and analysis of the plant's specific states at different time points and contributing to a deeper understanding of the mechanisms of plant growth. Furthermore, the method and device capture the dynamic changes in plant growth by combining the spatiotemporal fusion of multimodal data. Building on the traditional spatiotemporal convolutional network, an encoder-decoder structure is introduced to better capture the spatiotemporal variations in plant growth. The encoding stage extracts features, while the decoding stage reconstructs the growth process and enables future growth predictions. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 FIG1 is an application environment diagram of a process analysis method based on multimodal data in one embodiment;

[0059] Figure 2 1 is a flow chart of a process analysis method based on multimodal data in one embodiment;

[0060] Figure 3 A schematic diagram of a process flow of a plant growth analysis model according to an embodiment;

[0061] Figure 4 FIG1 is a flow chart of the steps of obtaining a historical multimodal dataset of a plant to be analyzed in one embodiment;

[0062] Figure 5 1 is a flow chart of a process analysis method based on multimodal data in yet another embodiment;

[0063] Figure 6 A schematic flow chart of a step of outputting plant future growth prediction data corresponding to future time steps of the plant to be analyzed through the plant growth analysis model in one embodiment;

[0064] Figure 7 A schematic diagram of a process flow of a plant growth analysis model according to an embodiment;

[0065] Figure 8 A schematic flow chart of a plant growth analysis model according to another embodiment;

[0066] Figure 9 1 is a flow chart of a training process of an adaptive graph neural network in one embodiment;

[0067] Figure 10 1 is a flow chart of a step of updating a hidden state of a node based on the node's own features and neighbor node features in an initial undirected graph in one embodiment;

[0068] Figure 11 A schematic flow chart of a plant growth analysis model according to another embodiment;

[0069] Figure 12 A schematic flow chart of a plant growth analysis model according to another embodiment;

[0070] Figure 13 A schematic flow chart of a plant growth analysis model according to another embodiment;

[0071] Figure 14 Schematic diagram of a flow chart of the steps of dynamically adjusting a preset threshold value in one embodiment;

[0072] Figure 15 A schematic flow chart of a plant growth analysis model according to another embodiment;

[0073] Figure 16 is a structural block diagram of a process analysis device based on multimodal data in one embodiment;

[0074] Figure 17 FIG. 4 is a diagram showing the internal structure of a gardening robot in one embodiment. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0076] With the continuous advancement of technology, the application of gardening robots and related technologies in the horticultural field has gradually increased. However, traditional gardening robots and related technologies still have many shortcomings in practical applications, especially in terms of intelligence, growth model construction, long-term dependency processing, and anomaly detection.

[0077] 1. Current plant-related robots are primarily focused on large-scale agricultural operations such as harvesting and picking. While they have achieved a high level of automation, their application in horticulture is relatively limited. These robots are often tasked with completing specific tasks, lacking intelligence and adapting to the complex, diverse, and personalized needs of horticultural environments. In contrast, horticultural activities encompass not only basic daily care such as watering and fertilizing, but also precise monitoring, prediction, and intervention of plant growth conditions. These demands require robots with a higher level of intelligence.

[0078] 2. Traditional gardening robots mostly follow preset parameters and rules for mechanized operations, such as watering and fertilizing according to a planned schedule and quantity. They rely primarily on machine vision and positioning technology to ensure operational accuracy. While this model achieves basic automation, it falls short of truly intelligent gardening management. They are unable to flexibly adjust their operating strategies based on real-time environmental conditions, individual plant differences, and growth status, and lack the ability to deeply understand and proactively adapt to plant growth processes.

[0079] 3. Traditional gardening robots lack the ability to continuously observe and record data to build plant growth models, nor can they use this data to predict growth or detect anomalies. These robots lack the ability to learn the growth patterns of specific plants. They are unable to scientifically predict plant growth processes through real-time monitoring and historical data analysis, nor can they promptly detect and respond to potential growth anomalies.

[0080] 4. Traditional smart agriculture often uses static or relatively simple dynamic models when constructing plant growth models, making it difficult to fully and accurately capture the spatiotemporal complexity of plant growth. These models typically only process a single or limited set of data types, such as environmental parameters and physiological indicators, and fail to fully leverage the complementary nature of multimodal data (such as spectral, imaging, and physiological and ecological data). This severely limits the models' understanding of plant growth processes and their predictive accuracy.

[0081] 5. Plant growth is a continuous process with significant time-dependence. An ideal growth model should possess a memory function, integrating historical information to improve the accuracy of future predictions. However, traditional smart agriculture technologies have significant shortcomings in handling long-term dependencies, generally lacking effective memory mechanisms to store and utilize historical data. This significantly reduces the model's predictive capabilities when faced with cyclical changes in plant growth and phase transitions.

[0082] 6. Despite extensive research on plant anomaly detection, practical applications still generally rely on fixed threshold methods. This approach ignores the fact that plant growth is influenced by multiple factors (such as season, variety, and growth stage). This makes it difficult to comprehensively account for various factors in threshold setting, resulting in high rates of false positives and false negatives. Furthermore, traditional technologies often lack deep fusion and analysis when processing real-time image and time-series data, making it difficult to accurately extract indicators reflecting key plant growth anomalies from multiple data sources. This reduces the accuracy of horticultural management tasks such as pest and disease early warning and nutritional imbalance diagnosis.

[0083] In summary, traditional gardening robots and related technologies suffer from numerous deficiencies in intelligence, growth model construction, long-term dependency processing, and anomaly detection. These limitations limit the effectiveness and performance of gardening robots in practical applications, and they fail to meet the demands of modern gardening for intelligence, precision, and efficiency. Therefore, developing gardening robots and related technologies that can comprehensively capture the spatiotemporal variations of plant growth, adaptively learn graph structures, incorporate memory mechanisms, and achieve accurate anomaly detection holds significant research value and application prospects.

[0084] The process analysis method based on multimodal data provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, specifically, in a gardening environment, a gardening robot 104 is pre-installed with multiple types of onboard sensors 102 and is factory-configured to perform a default task for reconstructing a plant growth process scenario. During use of the gardening robot, a user can invoke a pre-built plant growth analysis model by executing the plant growth process reconstruction task, thereby displaying the corresponding results. The onboard sensors 102 collect historical multimodal data and transmit this data to the gardening robot 104. After invoking the plant growth analysis model, the gardening robot 104 outputs plant growth history data corresponding to the historical multimodal dataset of the plant to be analyzed. A data storage system can store data that the gardening robot 104 needs to process. The data storage system can be integrated with the gardening robot 104 or located in the cloud or other network server. The onboard sensors 102 can include various image acquisition devices for acquiring image data to be processed, as well as various radar sensors for acquiring point cloud data to be processed, such as plant morphology.

[0085] In an exemplary embodiment, Figure 2 As shown in the figure, a process analysis method based on multimodal data is provided. Figure 1 The gardening robot 104 in FIG. 1 is used as an example to illustrate the process, including the following steps 202 to 206.

[0086] Step 202: Acquire a historical multimodal dataset of the plant to be analyzed.

[0087] Multimodal data includes spectral data, imaging data, and physiological and ecological data. A historical multimodal plant dataset refers to a collection of data recorded by a gardening robot during its continuous observation of plants. Specifically, multiple types of sensors, such as onboard sensor 102, are used to collect multidimensional data, including plant physiological status data, plant morphological characteristic data, and environmental impact data. These onboard sensors can include spectral analysis sensors, RGB cameras, infrared thermal imagers, membrane conductivity sensors, stereo vision sensors, lidar, and other sensors. They can also include air temperature and humidity sensors, soil temperature and humidity sensors, light intensity sensors, or pH sensors.

[0088] Step 204: call the plant growth analysis model.

[0089] Specifically, the plant growth analysis model may include a plant growth process reconstruction model, and the plant growth process reconstruction model may reconstruct the plant growth history based on historical multimodal data in a historical multimodal data set of the plant.

[0090] In an exemplary embodiment, please refer to Figure 3 , Figure 3 This is a flow chart of a plant growth analysis model according to an embodiment. For example, the plant growth analysis model according to this embodiment may also be referred to as a plant growth process reconstruction model.

[0091] In one embodiment, the plant growth process reconstruction model is mainly divided into two parts: a spatiotemporal encoder and a spatiotemporal decoder. The spatiotemporal encoder is responsible for extracting spatiotemporal features to obtain spatiotemporal feature vectors, while the spatiotemporal decoder is responsible for reconstructing the plant growth history based on these spatiotemporal feature vectors with spatiotemporal features, obtaining plant growth history data. This allows the plant growth analysis model to reconstruct the plant growth history from sowing to the current moment based on historical multimodal data, such as images, physiological parameters, and environmental data. Specifically, the plant multimodal dataset DataFrame is input into the spatiotemporal encoder of the plant growth analysis model and the plant growth history is reconstructed through the spatiotemporal decoder, thereby outputting plant growth history data. The plant growth history data includes growth process reconstruction text data, including detailed records of plant morphological changes (such as changes in morphological characteristics such as plant height, leaf area, and number of branches over time) and characteristic descriptions of physiological development stages (such as key periods of vegetative growth and reproductive growth), allowing users to intuitively review and analyze the status of plants at different stages. The plant growth history data also includes growth process reconstruction image data, which shows the growth status of plants at different time points and provides intuitive visual information.

[0092] In step 206, the plant multimodal dataset is input into the spatiotemporal encoder of the plant growth analysis model. The spatiotemporal encoder includes a first spatiotemporal convolutional layer, a second spatiotemporal convolutional layer, and a fully connected layer. The time series dataset of the plant to be analyzed is input into the first spatiotemporal convolutional layer to obtain plant growth time features, and the image dataset of the plant to be analyzed is input into the second spatiotemporal convolutional layer to obtain plant growth image features. The time series dataset and the image dataset are obtained by preprocessing the plant multimodal dataset.

[0093] Please also refer to Figure 3 The input of the plant growth analysis model is a plant multimodal dataset. First, the plant multimodal dataset is preprocessed and formatted. The plant multimodal dataset input to the spatiotemporal encoder of the plant growth analysis model includes two types of information:

[0094] ① Time series data: such as hourly, daily, and weekly data on plant physiological parameters, plant morphological characteristics, and environmental impact parameters, such as plant growth height, leaf area, chlorophyll content, soil moisture, and light intensity.

[0095] ② Image data: such as high-definition photos of plants taken regularly, which record the changes in visual characteristics such as plant morphology, color, pest and disease status over time.

[0096] Analysis reveals that the data exhibits a distinct spatiotemporal structure. A spatio-temporal convolutional neural network (ST-CNN) was selected to process this type of data. The ST-CNN layer extracts spatial patterns that evolve over time during plant growth. The ST-CNN layer consists of two layers: the first and second ST-CNN layers.

[0097] The plant multimodal dataset is formatted into two parts: the time series dataset Dt and the image dataset Di.

[0098] The shape of the time series dataset Dt is (T, Nf, 1), and the shape of the image dataset Di is a four-dimensional tensor (T, H, W, C), where:

[0099] T represents the number of time steps, corresponding to the number of observations; Nf represents the number of features, indicating the number of plant physiological state parameters, plant morphological characteristic parameters, and environmental influence parameters (such as plant height, leaf area, chlorophyll content, etc.) observed in each step; 1 represents the number of units, usually 1, indicating the observed value of a single parameter; H and W represent the image height and image width, indicating the image size; C represents the number of channels.

[0100] For example, the time steps are 0, 1, 2, and so on, where 0 represents the initial observation value. Users can set the interval between observations, such as "1 day," "5 days," "1 week," "1 month," and so on. For example, if the user selects "1 week," the gardening robot will collect data through sensors at the same time in the first, second, and third weeks, recording and constructing the data. It can be understood that shorter the interval between observations, the more consistent the data presentation, while shorter intervals result in less consistent data presentation.

[0101] Since the horticultural scene contains two different data types, time series data and image data, their dimensional structures are different and cannot be directly stacked into a four-dimensional tensor. Therefore, it is impossible to give a unified spatiotemporal convolutional layer formula to process these two types of data simultaneously.

[0102] Therefore, in one embodiment of the present application, the two types of data are processed separately by the first and second spatiotemporal convolutional layers. Their features, namely the plant growth time features and the plant growth image features, are then fused at appropriate locations through a fully connected layer. The following are the formulas for the spatiotemporal convolutional layers when processing the time series dataset and the image dataset, respectively:

[0103] 1) Time series dataset Dt:

[0104] The time series dataset Dt of the plant to be analyzed is input into the first spatiotemporal convolution layer. Specifically, one-dimensional convolution (for the time dimension) is used to process the time series dataset to obtain the plant growth time characteristics. The one-dimensional convolution kernel size is Kt, the step size is St, and no padding is used. The convolution operation formula is as shown in formula (1):

[0105] (1)

[0106] Among them, Xt' is the feature after convolution, that is, the plant growth time feature, and Wt is the one-dimensional convolution kernel.

[0107] 2) Image dataset Di

[0108] The image dataset Dt of the plant to be analyzed is input into the second spatiotemporal convolution layer. For the image dataset, a conventional spatiotemporal convolution kernel (taking both the temporal and spatial dimensions into account) is used for processing. For example, the spatiotemporal convolution kernel size is (Kt, Ks), the step size is (St, Ss), and no padding is used. The convolution operation formula is shown in Formula (2):

[0109] (2)

[0110] Among them, Xi' is the feature after convolution, that is, the plant growth image feature, and Wi is the spatiotemporal convolution kernel.

[0111] Step 208 : performing feature fusion on the plant growth time feature and the plant growth image feature through a fully connected layer to obtain a spatiotemporal feature vector.

[0112] For example, the time series dataset and image dataset are used as two independent input channels. Feature extraction is performed through their respective spatiotemporal convolutional layers, such as the first and second spatiotemporal convolutional layers, to obtain plant growth time features and plant growth image features. A fully connected layer then fuses these plant growth time features and plant growth image features to jointly serve the growth process reconstruction or future growth prediction tasks.

[0113] Assume that after processing by the spatiotemporal convolution layer and the self-attention layer, the feature map Xt of the time series dataset Dt has a shape of (T', Nf'), and the feature map Xi of the image dataset Di has a shape of (T', H', W', C'). To fuse these two types of features, a fully connected layer can be used to perform the following operations:

[0114] Flatten Xt into a one-dimensional vector Xt' with shape (T'×Nf',1);

[0115] Flatten Xi into a one-dimensional vector Xi' with the shape of (T'×H'×W'×C',1);

[0116] Concatenate Xt' and Xi' into a new vector Xti' with the shape of ((T'×Nf')+(T'×H'×W'×C'),1);

[0117] Input Xti' into the fully connected layer (including linear transformation and nonlinear activation function) to obtain the fused feature vector F: F=Activation(W⋅Xti'+b); where W is the weight matrix of the fully connected layer, b is the bias vector, and Activation is the nonlinear activation function (such as ReLU, Leaky ReLU, Sigmoid, etc.).

[0118] Step 210 , input the spatiotemporal feature vector into the spatiotemporal decoder of the plant growth analysis model to obtain plant growth history data; the plant growth history data includes growth process reconstructed text data and growth process reconstructed image data.

[0119] In this embodiment, a plant growth analysis model is invoked to process a historical multimodal dataset of the plant to be analyzed, thereby reconstructing the plant's growth history based on the historical multimodal data in the historical multimodal dataset, thereby outputting plant growth history data. The plant growth history data includes reconstructed text data and reconstructed image data of the growth process. The reconstructed text data and reconstructed image data of the growth process reflect detailed characteristics of key stages such as plant morphological changes and physiological development. Specifically, a spatiotemporal encoder processes time series data and image data of different data types and outputs a fused feature vector, namely a spatiotemporal feature vector F, whose shape is (D, 1), where D is the output dimension of the fully connected layer. The output spatiotemporal feature vector contains a high-level abstract representation of the combined features of the time series data and image data. This spatiotemporal feature vector can be used as input to a subsequent decoder or other prediction module for growth process reconstruction or future growth prediction tasks. Furthermore, by combining the spatiotemporal fusion of multimodal data to capture the dynamic changes in plant growth, an encoder-decoder structure is introduced on the basis of a traditional spatiotemporal convolutional network to better capture the spatiotemporal changes in plant growth. The encoding stage is responsible for extracting features, and the decoding stage is responsible for reconstructing the growth process and can make future growth predictions.

[0120] In the above-mentioned process analysis method based on multimodal data, the plant growth process is reconstructed through a plant growth analysis model based on the historical multimodal dataset of the plant to be analyzed, including detailed characteristics of key stages such as morphological changes and physiological development, so as to facilitate the observation and analysis of the specific states of plants at different time nodes. This not only helps to deeply understand the mechanism of plant growth, but also provides rich basic data for subsequent prediction and anomaly detection.

[0121] Specifically, the gardening robot can be configured to perform the task of reconstructing the scene of the plant growth process by default when it leaves the factory, which means that the robot can provide comprehensive data analysis services.

[0122] Please refer to Figure 4 , Figure 4 FIG. 1 is a flow chart of the steps of obtaining a historical multimodal dataset of a plant to be analyzed in one embodiment, wherein obtaining a historical multimodal dataset of a plant to be analyzed includes:

[0123] Step 402 , obtaining historical multi-dimensional data of the plant to be analyzed to obtain multimodal plant data; the historical multi-dimensional data at least includes plant physiological state data, plant morphological characteristic data, and environmental impact data.

[0124] Specifically, multi-dimensional data such as plant physiological status, morphological characteristics, and environmental impacts are collected through multiple types of sensors to achieve comprehensive plant growth monitoring.

[0125] The plant physiological state data reflects relevant information of multiple plant physiological state parameters. Table 1 exemplarily lists relevant information reflecting the collection of plant physiological state data, including plant physiological state parameters and problems reflected by different plant physiological state parameters.

[0126] Table 1 Related information for plant physiological status data collection

[0127]

[0128] Furthermore, among plant physiological parameters, a plant's chlorophyll content can reflect its photosynthetic capacity and nutritional status. A spectral analysis sensor can be used to collect the chlorophyll content of the plant being analyzed. For example, a spectral analysis sensor such as a SPAD (Soil and Plant Analyzer Development meter) or a high-resolution spectrometer can directly measure the chlorophyll content of the plant being analyzed. Alternatively, an RGB camera can be used to directly capture plant images of the plant being analyzed, and image processing techniques can be used to extract leaf color to indirectly measure the chlorophyll content. It is understood that spectral analysis sensors are typically designed for scientific research and professional agricultural applications and are relatively expensive. Therefore, when cost is a factor, an RGB camera can be used to measure the chlorophyll content of the plant being analyzed.

[0129] Among the plant physiological state parameters, the plant surface temperature distribution can reflect the plant surface temperature distribution and can be used to evaluate the leaf temperature and stem temperature. The sensor for obtaining the plant surface temperature can be an infrared thermal imager, which can obtain the distribution of the plant surface temperature.

[0130] Among plant physiological parameters, plant tissue conductivity can reflect the conductivity of the plant tissue being analyzed, thereby enabling the assessment of environmental stress. Membrane conductivity sensors can be used to measure plant tissue conductivity, while analog conductivity sensors can also be used to measure plant tissue conductivity. However, analog conductivity sensors are relatively expensive, making them an optional feature for gardening robots. Environmental stress refers to adverse abiotic and biotic conditions encountered by plants during their growth, which can limit their growth and development.

[0131] Furthermore, the plant morphological characteristic data reflects relevant information of multiple plant morphological characteristic parameters. Table 2 exemplarily lists relevant information reflecting the collection of plant morphological characteristic data, including plant morphological characteristic parameters, problems reflected by different plant morphological characteristic parameters, sensors used, and relevant remarks.

[0132] Table 2 Information related to plant morphological characteristics data collection

[0133]

[0134] Furthermore, among the plant morphological characteristic parameters, the size of the leaf area can affect the plant's light energy capture and gas exchange, that is, it reflects the plant's light energy capture and gas exchange capabilities. The plant's leaf area can be obtained through image data acquired by an RGB camera, for example, by extracting the leaf area data of the plant to be analyzed through image processing technology.

[0135] Among the plant morphological characteristic parameters, leaf color can reflect the chlorophyll content and health status of the plant to be analyzed. The leaf color of the plant can be obtained through image data acquired by an RGB camera, for example, by extracting the leaf color data of the plant to be analyzed through image processing technology.

[0136] Among the plant morphological characteristic parameters, leaf lesions and insect pest marks can reflect the occurrence of pests and diseases in the plant being analyzed. That is, leaf lesions and insect pest marks can be used to assess the occurrence of pests and diseases in the plant being analyzed. Leaf lesions and insect pest marks can be obtained from image data captured by an RGB camera. For example, image processing techniques can be used to extract leaf lesions and insect pest marks data for the plant being analyzed.

[0137] Among the plant morphological characteristic parameters, the stem diameter and length are related to the plant growth stage and biomass accumulation, that is, they can reflect the growth stage of the plant and also reflect the biomass accumulation of the plant. For example, a stereo vision system such as a binocular camera or a structured light camera can be used as a sensor to obtain the stem diameter and length of the plant to be analyzed. Alternatively, a lidar can be used as a sensor to obtain the stem diameter and length of the plant to be analyzed. It is understandable that since lidar equipment is usually designed for scientific research and professional agricultural applications and is relatively expensive, when considering cost factors, a stereo vision system can be used or optionally used as a sensor to obtain the stem diameter and length of the plant to be analyzed.

[0138] Among the plant morphological characteristic parameters, the number and angle of branches are related to light distribution and plant structure, that is, they can reflect the light distribution and structure of the plant. For example, a stereo vision system such as a binocular camera or a structured light camera can be used as a sensor to obtain the number and angle of branches of the plant to be analyzed. Alternatively, a lidar can be used as a sensor to obtain the number and angle of branches of the plant to be analyzed. It is understandable that since lidar equipment is generally designed for scientific research and professional agricultural applications and is relatively expensive, when considering cost factors, a stereo vision system can be used or optionally used as a sensor to obtain the number and angle of branches of the plant to be analyzed.

[0139] Furthermore, the environmental impact data reflects relevant information of multiple environmental impact parameters. Table 3 exemplarily lists relevant information reflecting the collection of environmental impact data, including environmental impact parameters, the issues reflected by different environmental impact parameters, the sensors used, and relevant remarks.

[0140] Table 3 Environmental impact data collection related information

[0141]

[0142] Furthermore, among the environmental impact parameters, air temperature and humidity can reflect the air temperature and humidity of the environment in which the plant to be analyzed is located, and can be used to assess whether suitable growth conditions are provided for the plant. The air temperature and humidity of the plant's environment can be detected using an air temperature and humidity sensor.

[0143] Among the environmental impact parameters, soil temperature and humidity can reflect the temperature and humidity of the soil where the plants to be analyzed are planted, thereby reflecting whether the irrigation and fertilization plans are reasonable and can be adjusted based on these parameters. The temperature and humidity of the soil where the plants to be analyzed are planted can be measured using a soil temperature and humidity sensor.

[0144] Among the environmental impact parameters, light intensity can reflect the intensity of light received by the plants to be analyzed, thereby reflecting whether the light intensity is reasonable and providing a basis for adjusting lighting conditions. The light intensity of the plants to be analyzed can be measured using a light intensity sensor.

[0145] Among environmental impact parameters, pH reflects the acidity or alkalinity of the plant or soil being analyzed, and thus indicates whether the growth environment of the plant being analyzed is suitable. pH is crucial for optimizing the plant's growth environment. pH sensors can be used to measure the pH value of the plant or soil being analyzed.

[0146] Step 404 : performing smoothing and normalization processing on the multimodal plant data to obtain processed multimodal plant data.

[0147] Exemplarily, filtering or smoothing techniques are applied to the collected plant multimodal data to remove noise. Denoising the collected plant multimodal data includes processing missing values and processing outliers.

[0148] For example, when processing missing values in plant multimodal data, when there is a set of plant height data in centimeters in the plant multimodal data, if there are missing values, the missing values in the plant height data can be processed by using methods such as linear interpolation, nearest neighbor interpolation, or mean filling. For example, for missing values, the average of the previous observation and the next observation is used to fill in the missing values:

[0149] [12, NaN, 18, 25, 20] → [12, (18+12) / 2, 18, 25, 20] = [12, 15, 18,25, 20]

[0150] Among them, [12, NaN, 18, 25, 20] represents the plant growth height data, NaN represents a missing value, and it is filled with the average value 15 of the observation value 12 before NaN and the observation value 18 after NaN, thereby completing the processing of missing values in the plant multimodal data. The processed plant growth height data is [12, 15, 18, 25, 20].

[0151] For example, when processing outliers in plant multimodal data, when there is a set of soil moisture data (unit: %) in the plant multimodal data, the outliers are identified and processed using the Z-score method. The standard deviation multiple (Z-score) of each data point relative to the sample mean is calculated. For example, data points with an absolute value exceeding 3 are considered outliers. For example:

[0152] Soil moisture data: [20, 22, 24, 26, 300, 28, 30];

[0153] Get the mean of the soil moisture data: μ = (20 + 22 + 24 + 26 + 28 + 30) / 6 = 24.5;

[0154] Get the standard deviation of the soil moisture data: σ = sqrt(((20-24.5)^2 + ... + (30-24.5)^2) / 6) ≈ 1.581;

[0155] Soil moisture data were processed using the Z-score method: [(20-24.5) / 1.581, ... , (30-24.5) / 1.581];

[0156] Get outliers: Since the Z-score of data 300 in the soil moisture data is much greater than 3, replace it with a reasonable value. For example, the median of the adjacent data is used for filling and replacement as follows:

[0157] [20, 22, 24, 26, 28, _300_, 28, 30]→[20, 22, 24, 26, 28, 27, 28, 30]

[0158] This completes the processing of outliers in the plant multimodal data, that is, the processed soil moisture data is [20, 22, 24, 26, 28, 27, 28, 30].

[0159] And, in one embodiment, different modality data in the plant multimodal data are normalized or standardized to ensure the consistency and comparability of the data.

[0160] For example, when normalizing multimodal plant data, take light intensity data as an example. For light intensity data (unit: lux), it is converted to the range [0, 1] to ensure comparability between different data sets. For example, the min-max normalization formula is used:

[0161] For each light intensity data value x:

[0162] New value of light intensity data = (x - min(light intensity data)) / (max(light intensity data) - min(light intensity data)) (3)

[0163] Where min(light intensity data) represents the minimum value of the light intensity data; max(light intensity data) represents the maximum value of the light intensity data. That is, for each light intensity data value x, the updated light intensity data value, i.e., the new value, can be obtained by formula (3).

[0164] For example, when performing standardization on plant multimodal data, temperature data is taken as an example.

[0165] The temperature data (unit: °C) is Z-score normalized to have a normal distribution with a mean of 0 and a standard deviation of 1. The processing method is as follows:

[0166] For each temperature data value x:

[0167] New value of temperature data = (x - μ(temperature data)) / σ(temperature data) (4)

[0168] Among them, μ(temperature data) is the mean of the temperature data, and σ(temperature data) is the standard deviation of the temperature data. That is, for each temperature data value x, the updated temperature data value, i.e., the new temperature data value, can be obtained by formula (4).

[0169] Through the above-mentioned denoising and standardization processing of the multimodal plant data, the preprocessing of the multimodal plant data is achieved to obtain the processed multimodal plant data.

[0170] Step 406 , performing time stamp synchronization processing on the processed multimodal plant data to obtain synchronized multimodal plant data; the timestamps of the synchronized multimodal plant data at the same time point are aligned.

[0171] Specifically, by performing timestamp synchronization processing on the processed multimodal data and performing interpolation processing when there is data missing at a certain time point in the data, the time series of the processed multimodal plant data can be aligned, thereby obtaining synchronized multimodal data.

[0172] First, timestamp synchronization is used to ensure that all processed multimodal data have accurate timestamps and that data at the same time point are correctly matched.

[0173] For example, when the timestamps of humidity data and light intensity data in the processed multimodal data are slightly offset, they need to be aligned to the same time axis.

[0174] For example, when the timestamp of humidity data is ["2024-04-15 09:30:00", "2024-04-15 10:00:00"] and the timestamp of light intensity data is ["2024-04-15 09:28:30", "2024-04-15 10:01:45"], you can round them to the nearest minute to achieve rough alignment.

[0175] Furthermore, if the data at a certain time point in the processed multimodal data is missing, the data at that time point can be estimated using an interpolation method.

[0176] For example, at a certain time point t, if the light intensity data is missing, and the adjacent observation time points are known to be t1 and t2, and the light intensity values corresponding to the adjacent observation time points are y1 and y2, then the light intensity value at the time point t can be obtained by interpolation formula (5), thereby estimating the light intensity data at the time point t:

[0177] y(t) = y1 + ((y2 - y1) / (t2 - t1)) * (t - t1) (5)

[0178] Step 408 : Associating each data in the synchronized multimodal plant data with the corresponding plant individual identifier to obtain traceable multimodal plant data.

[0179] For example, each plant individual has a plant individual ID, called a plant individual identifier. For example, Table 4 is a data table containing plant growth data, in which each row represents the plant individual identifier corresponding to a measurement, the time of measurement, and the measured plant growth height corresponding to the plant. The data table has fields such as PlantID, MeasurementTime, and Height, where PlantID represents the plant individual identifier, MeasurementTime represents the time of measurement, and Height represents the plant growth height. By associating each data in the synchronized multimodal plant data with the corresponding plant individual identifier, ensuring that each observation data is associated with the corresponding plant individual representation, traceable multimodal plant data can be obtained, such as:

[0180] Table 4 Plant growth data table

[0181]

[0182] For example, the plant with individual identification as P1 has a plant growth height of 15.3 cm obtained by the sensor on April 14, 2024; the plant growth height of the plant obtained by the sensor on April 15, 2024 is 16.1 cm, thereby tracking and analyzing the growth of the plant with individual identification as P1.

[0183] Step 410 : performing data integration on the traceable multimodal plant data to obtain a plant multimodal dataset.

[0184] Specifically, a plant multimodal dataset is constructed by merging plant physiological status data, plant morphological characteristic data, and environmental impact data into the same dataset.

[0185] In one embodiment, the multimodal plant data includes the following data:

[0186] Plant physiological status data: chlorophyll content (Chl), plant tissue electrical conductivity (Cond);

[0187] Plant morphological characteristic data: plant growth height (Height), number of leaves (LeafCount);

[0188] Environmental impact data: temperature (Temp), humidity (Humidity), and light intensity (Light).

[0189] For example, these data are represented by a data frame, as shown in Table 5, where each row represents a specific time point, each column represents a specific measurement value, and each record has an associated plant individual ID.

[0190] Table 5 Data framework of multimodal plant data

[0191]

[0192] In this data frame, all the measurements for each plant ID (PlantID) at each time point (TimeStamp) are shown. This structure makes it easy to analyze and process the data and build plant growth models.

[0193] In this way, we can not only track the changes of each individual plant over time, but also analyze the differences between different plants and how environmental factors affect plant growth. This integrated dataset provides a solid foundation for subsequent deep learning analysis and the establishment of personalized care models.

[0194] Specifically, you can load various data from different data sources (such as CSV (Comma-Separated Values) files, database tables, API (Application Programming Interface) interfaces, etc.), pre-process them separately, and then merge the processed data into a unified dataset.

[0195] For example, Pandas (an open source data analysis and processing library for the Python programming language) is used to merge the preprocessed growth height, soil moisture, and light intensity data into a plant multimodal dataset DataFrame.

[0196] In this embodiment, multimodal plant data of the plants to be analyzed are collected, and the multimodal data are aligned and integrated, so that the multimodal plant data collected by multiple types of sensors, including at least plant physiological status data, plant morphological characteristic data and environmental impact data, can be integrated into the same data framework to obtain a unified data set, which is convenient for the final analysis of the growth of individual plants, and can also analyze the differences between different plants and how environmental factors affect plant growth, thereby providing a solid foundation for the subsequent analysis of plant growth through plant growth analysis models.

[0197] In an exemplary embodiment, inputting the spatiotemporal feature vector into the spatiotemporal decoder of the plant growth analysis model to obtain plant growth history data includes:

[0198] The spatiotemporal feature vector is input into the spatiotemporal deconvolution layer of the spatiotemporal decoder. The temporal resolution and spatial resolution of the spatiotemporal feature vector are increased by the spatiotemporal deconvolution layer to obtain the plant growth process data.

[0199] Please continue to refer to Figure 3 The input of the spatiotemporal decoder is the spatiotemporal feature vector output by the spatiotemporal encoder. The spatiotemporal feature vector serves as the basis for the spatiotemporal decoder to reconstruct the plant growth process and make future predictions. Its shape is (D, 1).

[0200] Corresponding to the spatiotemporal convolutional layers in the spatiotemporal encoder, the spatiotemporal decoder uses spatiotemporal deconvolutional layers to further process features and reconstruct the plant growth process. These layers can generate detailed growth history images based on the features extracted in the encoding phase.

[0201] The spatiotemporal deconvolution layer is the core component of the spatiotemporal decoder, corresponding to the spatiotemporal convolution layer in the spatiotemporal encoder. Through deconvolution, the spatiotemporal decoder can gradually increase the spatial and temporal resolution of the feature map, thereby recovering more detailed information about the plant growth process.

[0202] The deconvolution operation expands the size of the feature map by inserting zero values (upsampling) and performing convolution operations. By choosing an appropriate convolution kernel size and stride, the feature map can be gradually restored to a size similar to the original input.

[0203] For the encoding features output by the spatiotemporal encoder, that is, the spatiotemporal feature vector F, the spatiotemporal deconvolution operation can be expressed by formula (6):

[0204] (6)

[0205] in:

[0206] Upsample(F) performs an upsampling operation on the spatiotemporal feature vector F, expanding its size to the target dimensions T, H, W by inserting zeros, where ∗ represents a convolution operation. W is the spatiotemporal deconvolution kernel, whose shape is (Kt, Ks, C', C), where Kt and Ks are the kernel sizes in the temporal and spatial dimensions, respectively, C' is the number of channels in the input feature map, and C is the number of channels in the output feature map.

[0207] Optionally, in addition to the spatiotemporal deconvolution layers, the spatiotemporal decoder can also include other auxiliary layers, such as batch normalization layers (for accelerating training and improving model stability) and activation function layers (such as ReLU or Sigmoid, for introducing nonlinear characteristics). These layers can be added and adjusted according to specific task requirements.

[0208] And further optionally, when reconstructing text data during the output growth process, a natural language processing library can be introduced to assist in the output.

[0209] After the spatiotemporal feature vector is processed by the spatiotemporal deconvolution layer of the spatiotemporal decoder, the spatiotemporal decoder outputs the plant growth history data, which includes the reconstructed text data and the reconstructed image data of the growth process:

[0210] 1) Growth process reconstruction text data: The final output of the spatiotemporal decoder is the reconstructed growth process, including detailed records of morphological changes and descriptions of physiological development stages.

[0211] Morphological Change Record: This output provides information about plant morphological changes during growth, such as changes in plant height, leaf area, and branch number over time. This information can be obtained by post-processing and analyzing the feature maps output by the spatiotemporal decoder, for example, by calculating pixel values or applying threshold segmentation.

[0212] Physiological Development Stage Description: In addition to morphological changes, the decoder can also output characteristic descriptions of plants at different physiological development stages. This can be achieved by training the model to identify characteristics of key periods (such as vegetative growth and reproductive growth). These descriptions can help users better understand the growth process and physiological state of plants.

[0213] 2) Reconstructed Growth Image Data: In addition to the textual descriptions described above, the decoder also outputs a sequence of reconstructed images of the same size as the input image. These images depict the plant's growth status at different points in time, providing intuitive visual information. Users can analyze the plant's growth history and changing trends by observing these images.

[0214] The spatiotemporal deconvolution layer is the core component of the spatiotemporal decoding layer, which corresponds to the spatiotemporal convolution layer in the spatiotemporal encoder. By introducing the spatiotemporal deconvolution layer into the spatiotemporal decoder, not only is the size of the feature map restored, but features are also further extracted and integrated to generate more accurate growth history images. These images can reflect key information such as the morphological changes and growth rate of the plant at different time points. In this embodiment, through the deconvolution operation, the spatiotemporal decoder can gradually increase the spatial and temporal resolution of the feature map, thereby recovering more detailed growth process information.

[0215] In order to enable the gardening robot to predict the growth status of plants in the future, in an exemplary embodiment, Figure 5 As shown, the process analysis method based on multimodal data also includes:

[0216] Step 502: Obtain the number of future time steps.

[0217] Step 504: The plant growth analysis model further outputs plant future growth prediction data corresponding to the future time steps of the plant to be analyzed; the plant future growth prediction data includes key growth indicator prediction text data and growth status prediction image data.

[0218] The future time step number T_future is input into the model inference process of the plant growth analysis model. The trained plant growth analysis model can predict the plant growth status at future moments, so that the plant growth analysis model also outputs future growth prediction data, including key growth indicator prediction text data and growth status prediction image data. Specifically, in this embodiment, the plant growth analysis model can be referred to as a plant future growth prediction model. The plant future growth prediction model uses a plant growth process reconstruction model that learns the basic characteristics of plant growth as its foundation layer. Based on the plant growth process reconstruction model, it further learns and utilizes historical and real-time data to predict future plant growth trends, including important indicators such as growth rate, biomass accumulation, and flowering period.

[0219] The multimodal data-based history analysis method of this embodiment uses existing historical data and real-time monitoring data to predict plant growth conditions and trends over the next period of time, including key indicators such as growth rate, biomass accumulation, and flowering period. By predicting key growth indicators such as growth rate, biomass accumulation, and expected maturity or flowering period, the plant growth analysis model can provide growers with forward-looking information, enabling them to make more scientific and reasonable management decisions such as planting planning, resource allocation, and harvesting arrangements, thereby improving the efficiency and quality of horticultural production. In addition, the hierarchical and progressive learning strategy of the plant future growth prediction model based on the plant growth process reconstruction model enables the plant future growth prediction model to be further improved on the basis of the plant growth process reconstruction model, thereby gaining a deeper understanding of the complex process of plant growth. Furthermore, since the plant future growth prediction model is constructed based on the plant growth process reconstruction model, the features extracted by the plant growth process reconstruction model can be reused. This reuse mechanism reduces repeated feature extraction work and improves computational efficiency. By adding layers to the existing model to create a new model, it is possible to avoid building a completely new complex model from scratch, which helps to control the complexity of the model and reduce the risk of overfitting that may be caused by an overly complex model. When there is new data or new research findings, only the relevant model layers need to be updated instead of the entire model, which makes model maintenance and updating easier and more efficient. Since the underlying structure is shared between the plant future growth prediction model and the plant growth process reconstruction model, a lot of time and computing resources can be saved during training, which is especially important when the data set is large or computing resources are limited. In practical applications, certain model layers can be enabled or disabled as needed, thereby providing customized solutions without the need to redesign or deploy the entire system.

[0220] In an exemplary embodiment, Figure 6 As shown, the plant growth analysis model also outputs the plant future growth prediction data corresponding to the future time steps of the plant to be analyzed, including steps 602 and 604. Specifically, please refer to Figure 7 , Figure 7 This is a flow chart of the plant growth analysis model of this embodiment. For example, the plant growth analysis model of this embodiment can also be called a plant future growth prediction model. Specifically, step 602 and step 604 include:

[0221] In step 602, the spatiotemporal feature vector is input into the adaptive graph neural network in the plant growth analysis model to obtain an adaptive graph structure. The spatiotemporal feature vector is obtained by processing the historical multimodal dataset through the spatiotemporal encoder of the plant growth analysis model. The adaptive graph structure is used to reflect the intrinsic relationship between different data modalities.

[0222] Step 604: input the adaptive graph structure and the future time steps into a spatiotemporal decoder in the plant growth analysis model to obtain future plant growth prediction data.

[0223] For example, the input of the plant growth analysis model is a plant multimodal dataset. As mentioned above, in this embodiment, the plant growth analysis model can also be called a plant future growth prediction model. Figure 7 and Figure 3 The future growth prediction model of plants is based on the plant growth process reconstruction model with some new layers. Specifically, the future growth prediction model of plants is based on the plant growth process reconstruction model with an adaptive graph neural network, and the future time steps are added as input, wherein the adaptive graph neural network module is after the spatiotemporal encoder and before the spatiotemporal decoder. Exemplarily, the spatiotemporal feature vector obtained by the spatiotemporal encoder processing the historical multimodal dataset is input into the adaptive neural network in the plant growth analysis model to obtain an adaptive graph structure for reflecting the intrinsic relationship between different data modalities, and the adaptive graph structure and the future time steps are input into the spatiotemporal decoder in the plant growth analysis model to obtain the future growth prediction data of plants. At the same time, the spatiotemporal feature vector obtained by the spatiotemporal encoder processing the historical multimodal dataset is directly input into the spatiotemporal decoder to obtain the plant growth history data.

[0224] Specifically, the spatiotemporal encoder extracts features from raw data, such as preprocessed historical multimodal datasets. These features may include the spatial layout of plants, growth rates, and environmental factors. A graph neural network (GNN) is then used to further process these features and generate higher-level predictions. These predictions are then used as input to the spatiotemporal decoder for future growth predictions, resulting in plant growth prediction data as the output of the plant growth prediction model. This plant growth prediction data includes textual data for key growth indicator predictions and image data for growth status predictions. The textual data includes detailed morphological changes (e.g., changes in plant height, leaf area, and branch number over time) and descriptions of physiological development stages (e.g., key periods of vegetative and reproductive growth), allowing users to intuitively review and analyze the plant's status at different stages. The image data for growth status predictions provides intuitive visual information about the plant's growth at different points in time.

[0225] The adaptive graph neural network in this embodiment can be seen as a bridge connecting the spatiotemporal encoder and the spatiotemporal decoder. The adaptive graph neural network dynamically generates a graph structure based on the characteristics of the input data itself. This method is not based on fixed rules or assumptions, but uses a data-driven approach to learn the connection strength and direction between nodes. In the scenario of plant growth prediction, this means that based on multimodal information such as plant growth data, environmental data, and real-time monitoring data, a graph structure that can reflect the actual situation of plant growth can be adaptively constructed to make more reasonable predictions, thereby converting the output of the spatiotemporal encoder into a form that is more suitable for processing by the spatiotemporal decoder.

[0226] Traditional graph neural networks often only focus on current information when processing sequential data, ignoring the accumulation and impact of historical information. However, in plant growth forecasting, historical data often contains rich growth trends and cyclical patterns, which are crucial for accurately predicting future growth conditions.

[0227] Therefore, in order to better capture long-term dependencies, a memory unit is introduced into the adaptive graph neural network so that the plant growth analysis model can remember historical information and use it for future predictions. In an exemplary embodiment, Figure 8 As shown, the adaptive graph neural network further includes a memory unit layer, and the process analysis method based on multimodal data further includes steps 802 and 804:

[0228] In step 802, dependency information extraction is performed on the plant growth time features through the memory unit layer to obtain long-term dependency information. The plant growth time features are processed by the first spatiotemporal convolutional layer in the spatiotemporal encoder to obtain the time series dataset of the plant to be analyzed. The long-term dependency information is used to represent the historical growth trends and patterns of the plant to be analyzed.

[0229] In step 804 , the long-term dependency information is input into the adaptive graph neural network as an additional node feature to obtain an adaptive graph structure.

[0230] Specifically, for data with distinct time series characteristics (such as continuous environmental monitoring data and changes in plant physiological indicators over time), a separate memory unit layer is added after the adaptive graph neural network. The memory unit layer is responsible for processing the time series data, extracting long-term dependency information, and inputting this information into the adaptive graph neural network as additional node features. More specifically, the memory unit layer extracts dependency information from the temporal characteristics of plant growth, which are obtained by processing the time series dataset of the plant being analyzed through the first spatiotemporal convolutional layer in the spatiotemporal encoder.

[0231] In this embodiment, and to better capture long-term dependencies, a memory mechanism is introduced into the adaptive graph neural network. This allows the plant growth analysis model to remember historical information and use it for future predictions. Specifically, memory units are used to capture and store historical information, allowing it to be better utilized for predictions. Furthermore, the addition of a separate memory unit layer reduces computational complexity and is more suitable for lightweight applications.

[0232] In one embodiment, Figure 9 As shown in Figure 2, the training process of the adaptive graph neural network is as follows:

[0233] Step 902, constructing an initial undirected graph; the initial undirected graph is used to represent an undirected graph in which potential connections exist between nodes and the initial connection weight is less than an initial threshold; the nodes include plant growth parameters and the number of environmental variables.

[0234] Step 904: Use the spatiotemporal feature vector as input of the initial undirected graph, where each spatiotemporal feature vector corresponds to a node of the initial undirected graph.

[0235] Step 906: Update the hidden state of the node based on the node's own features and neighbor node features in the initial undirected graph.

[0236] Step 908: Update the edge weights between nodes using the hidden states of the nodes and global context information to obtain an adaptive graph neural network.

[0237] Specifically, the input to the adaptive graph neural network is the fused spatiotemporal feature vector F output by the spatiotemporal encoder, with a shape of (D, 1). Each observation point or feature in the encoding phase is considered a node in the graph. For example, environmental parameters at each time point are combined into a node, different physiological indicators of each plant constitute their own node, and image pixels at specific locations can also serve as nodes. Node features contain detailed information about their respective data types, such as numerical environmental measurements, pixel color, or texture features. The specific training process is as follows:

[0238] a. Construct an initial undirected graph

[0239] First, a completely undirected or sparse graph is constructed, in which all nodes are potentially connected, but the initial connection weights (connection strengths) are set to be less than an initial threshold, where the initial threshold is a small random value or a uniform initialization constant. This setting allows the adaptive neural network to update these weights through learning during subsequent training.

[0240] b. Learning edge weights

[0241] Using a specific learning mechanism (such as self-attention mechanism, edge prediction module or graph autoencoder) to learn the connection strength between nodes, includes the following steps:

[0242] 1) Message Passing: Spatiotemporal feature vectors are used as input to an initial undirected graph, where each spatiotemporal feature vector corresponds to a node in the initialized undirected graph. The hidden state of each node in the initial undirected graph is updated based on its own features and the features of its neighboring nodes. That is, each node computes a message based on its own features and the features of its neighboring nodes. Optionally, message passing can be implemented through multiple GNN (Graph Neural Network) message passing layers, such as a Graph Convolutional Network (GCN) or a Graph Attention Network (GAT). Equation (7) shows how to obtain the hidden state of node ν at layer l+1.

[0243] (7)

[0244] Among them, h u (l) is the hidden state of node u in layer l; N(v) represents the set of neighbor nodes of node v; Aggregate represents an aggregation function, such as sum, average or weighted sum; W (l) is the learnable weight matrix of layer l.

[0245] 2) Edge weight update: The edge weights between nodes are updated using the hidden states of the nodes and global context information to obtain an adaptive graph neural network. Specifically, based on the information of the interactions between nodes during the message passing process and possible global context information, the edge weights are calculated using an edge update function (such as a multi-layer perceptron (MLP)) to obtain an adaptive neural network. The edge weights reflect the correlation and influence between nodes in predicting plant growth. The edge weights are obtained as shown in formula (8):

[0246] w uv = f edge (h u , h v , g) (8)

[0247] Among them, w uv is the edge weight from node u to node v; f edge is the edge update function, which includes nonlinear transformation (such as ReLU) and normalization steps; g is the global context information.

[0248] In this embodiment, a graph neural network is used to model the complex interactions within plants. While the graph structure is not predefined, it is adaptively learned based on multimodal data. Specifically, the edges (connections between nodes) and the corresponding connection strengths and directions in the graph structure are no longer predefined, but are automatically generated by the adaptive graph neural network through a learning process, thereby providing a more comprehensive understanding of the internal mechanisms of plant growth. The adaptive graph neural network obtained through the above-mentioned adaptive graph neural network learning process and the adaptive graph structure obtained based on the adaptive graph neural network can not only reflect the intrinsic correlations between different data modalities, but also reveal their importance in the task of plant growth prediction. The adaptive graph structure itself provides an intuitive model explanation, revealing which environmental factors, physiological states, or image features are most critical for growth prediction. This helps guide management measures such as precision irrigation, fertilization, or pest control by gardening robots or users.

[0249] In an exemplary embodiment, Figure 10 As shown, step 906 includes step 1002 and step 1004, wherein:

[0250] Step 1002 : embedding a memory unit during the node update process so that each node is attached with a corresponding memory state; the memory state is used to store historical information of the node.

[0251] Step 1004: In each message passing iteration, the node receives information from neighbor nodes of the node and interacts with the node's own memory state to update the node's memory state and hidden state.

[0252] For example, LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit) can be used as memory units. Both memory units effectively handle long-term dependencies in sequence data, preserving and transmitting historical information through internal state. LSTM has a more complex structure, including a forget gate, input gate, and output gate, making it better suited for complex sequence data. GRU, on the other hand, has a simpler structure and higher computational efficiency, making it suitable for scenarios with high real-time requirements.

[0253] Specifically, memory units are directly embedded in the node update process within the adaptive graph neural network. This allows each node to maintain not only conventional node features, such as hidden states, but also a corresponding memory state, which stores the node's historical information. During each message-passing iteration, a node not only receives information from its neighbors but also interacts with its own memory state, updating it and generating new node features. While computationally intensive, this approach allows for more tightly integrated node updates with historical information, potentially resulting in better predictions.

[0254] In order to enable the gardening robot to monitor abnormal conditions during plant growth in real time, in an exemplary embodiment, Figure 11 As shown, the process analysis method based on multimodal data also includes:

[0255] Step 1102: Acquire a single-frame plant image of the plant to be analyzed.

[0256] Step 1104 , using the real-time anomaly detection module in the plant growth analysis model, performs leaf feature extraction processing on the single-frame plant image to obtain leaf features corresponding to the plant to be analyzed.

[0257] Step 1106: Determine whether there are any abnormalities in the leaves of the plant to be analyzed based on the leaf characteristics.

[0258] Step 1108: When there is an abnormality in the leaves of the plant to be analyzed, an immediate abnormality warning signal and an immediate abnormality detail report for the plant to be analyzed are output.

[0259] Please also refer to Figure 12 , Figure 12 This is a flow chart of the plant growth analysis model of this embodiment. For example, the plant growth analysis model of this embodiment can also be called a plant growth anomaly detection model.

[0260] For example, the model is initialized and data is prepared. For example, real-time data on plant growth is collected through sensors, including environmental impact data, plant morphological characteristics, and plant physiological status data. This data includes both text and image data. The text data includes, for example, environmental parameters and plant growth data, while the image data includes plant images at corresponding time points. Furthermore, an object detection or segmentation algorithm is pre-trained for use in the real-time anomaly detection module.

[0261] When performing instant anomaly detection, a single-frame image is first input, that is, the single-frame plant image data collected by the sensor in real time is input into the instant anomaly detection module; then image processing and feature extraction are performed. Specifically, through the trained target detection or segmentation algorithm, a detailed visual inspection is performed on the single-frame plant image, so as to perform leaf feature extraction processing, such as extracting leaf color, shape, texture and other features to obtain the leaf features corresponding to the plant to be analyzed; then abnormality judgment and early warning are performed, that is, based on the extracted leaf features, it is judged whether there are abnormal phenomena on the leaves, such as color variation, lesions or traces of insect infestation, etc.; if abnormal phenomena are found in the leaves of the plant to be analyzed, an immediate early warning signal is immediately generated, and an immediate abnormality detail report containing abnormal details is sent, such as the location of the abnormality and the possible cause.

[0262] In this embodiment, by analyzing image data from a single point in time, abnormal plant conditions at that specific moment can be effectively identified. These abnormalities include, but are not limited to, leaf color variations, leaf lesions, and signs of insect infestation. Through detailed visual inspection of these images, plant health can be quickly and accurately determined, and timely alerts can be issued, ensuring timely abnormality detection.

[0263] In one embodiment, as Figure 13 As shown, the process analysis method based on multimodal data also includes:

[0264] Step 1302 : When there is no abnormality in the leaves of the plant to be analyzed, obtain plant time series data of the current time and the plant to be analyzed.

[0265] Step 1304: Outputting the plant growth prediction data corresponding to the current time of the plant to be analyzed through the spatiotemporal decoder in the plant growth analysis model.

[0266] Step 1306: Input the current plant growth prediction data and the plant time series data into the time series anomaly analysis module in the plant growth analysis model for comparison.

[0267] Step 1308: When the difference between the plant's current growth prediction data and the plant's time series data exceeds a preset threshold, it is confirmed that the plant to be analyzed has growth abnormalities, and a time series abnormality warning signal and a time series abnormality detail report are generated for the plant to be analyzed.

[0268] Step 1310 , when the difference between the current plant growth prediction data and the plant time series data does not exceed a preset threshold, confirm that the plant to be analyzed has no growth abnormality, and generate good plant growth status information for the plant to be analyzed.

[0269] Exemplarily, when performing plant growth prediction, when there is no abnormality in the leaves of the plant to be analyzed, the time series data of the current time T and the plant to be analyzed are obtained, and the current time T is input into the spatiotemporal decoder in the trained plant growth analysis model. More specifically, the current time is input into the spatiotemporal decoder in the trained plant future growth prediction model, and the spatiotemporal decoder outputs the plant current time growth prediction data of the plant to be analyzed corresponding to the current time based on the current time T and the learned growth pattern, including growth rate, growth height, stem diameter and length, number and angle of branches, etc.; then a time series anomaly analysis is performed: the plant growth time series data (such as historical growth records) within a period of time, i.e., the time series data, and the plant growth prediction at the current time, i.e., the plant current time growth prediction data, are input into the time series anomaly analysis module for comparison; further, the time series anomaly analysis module evaluates the growth of the plant by comparing the actual observation data, i.e., the plant time series data, with the growth prediction data. If it is found that the plant growth rate is abnormal, the plant growth height does not meet expectations, the stem diameter and length changes are abnormal, or the number and angle of the branches are inconsistent with the prediction, etc., that is, when the difference between the plant current time growth prediction data and the plant time series data exceeds the preset threshold, it is considered that there is a growth abnormality, and an early warning signal and an abnormality details report are generated, thereby realizing abnormality assessment and early warning, and when the difference between the plant current time growth prediction data and the plant time series data does not exceed the preset threshold, it is confirmed that there is no growth abnormality in the plant to be analyzed, and good plant growth status information is generated for the plant to be analyzed, wherein the preset threshold can be a numerical value obtained based on experience that reflects the abnormal plant growth rate, the growth height does not meet expectations, the abnormal changes in the stem diameter and length, or the number and angle of the branches are inconsistent with the prediction. Further, when the difference between the plant current time growth prediction data and the plant time series data does not exceed the preset threshold, it is confirmed that there is no growth abnormality in the plant to be analyzed, and good plant growth status information is generated for the plant to be analyzed. That is, the process analysis method based on multimodal data in this embodiment can combine the judgment results of the immediate anomaly detection module and the time series anomaly analysis module to comprehensively evaluate the growth status of the plant. If no anomaly is detected by both modules, the user will be fed back information that the plant is in good growth condition; if an anomaly is detected, a corresponding early warning signal and an anomaly detail report will be sent so that the user can take timely intervention measures.

[0270] In this embodiment, time-series anomaly analysis enables, in addition to single-point anomaly detection, the long-term monitoring of plant growth trends, combined with a series of chronologically collected image data. This includes assessing plant growth rate, measuring growth height, monitoring changes in stem diameter and length, and observing the number and growth angle of branches. Continuous analysis of this time-series data can reveal potential growth problems or developmental deviations, allowing the gardening robot or user to take timely intervention measures. The plant growth anomaly detection model in this embodiment integrates real-time image analysis and time-series data analysis. By combining analysis of single-frame images (real-time sensor data) and time-series data, it enables real-time monitoring and periodic assessment of growth anomalies, ensuring the timeliness and accuracy of anomaly detection. By accurately identifying and addressing key abnormal indicators in plant growth, it contributes important technical support and a comprehensive solution to the development of smart gardening.

[0271] Furthermore, in this embodiment, anomaly detection is performed at the current moment and during the growth process by fusing real-time images and time series data. That is, by accurately identifying and processing key abnormal indicators in the plant growth process, abnormal conditions in the plant growth process are monitored in real time, and the growth pattern expected by the plant growth analysis model is compared with the actual observation data. This allows for timely identification of growth anomalies, occurrence of pests and diseases, or environmental stress, providing users with early warning information, thereby contributing important technical support to the development of smart gardening.

[0272] Furthermore, in order to ensure that the plant growth analysis model can flexibly respond to differences in plant growth patterns caused by factors such as seasons, varieties, and growth stages, and adapt to changes in plant growth conditions in real time, in an exemplary embodiment, Figure 14 As shown, the process analysis method based on multimodal data further includes a step of dynamically adjusting the preset threshold, specifically including step 1602 and step 1604:

[0273] Step 1402 , based on an external data source or an internal state of a plant growth analysis model, estimate and obtain key factor data affecting plant growth; the internal state of the plant growth analysis model is used to represent the state of data related to the key factor data in the multimodal plant dataset after being processed by the plant growth analysis model.

[0274] Step 1404: Dynamically adjust the preset threshold based on the key factor data.

[0275] To ensure that the model can flexibly respond to differences in plant growth patterns caused by factors such as seasons, varieties, and growth stages, and adapt in real time to changes in plant growth conditions, in one embodiment, a multimodal data-based history analysis method is proposed to dynamically adjust the threshold of the plant growth anomaly detection model. The specific implementation method is as follows:

[0276] 1. Design a preset threshold function that can be dynamically adjusted

[0277] The design of the preset threshold function needs to consider the key factors that affect plant growth, such as season, planting variety, growth stage, etc. These factors will affect the fluctuation range of the normal growth pattern. That is, the values of factors that affect plant growth, such as season, variety, growth stage, etc., are the key factor values. For example, formula (9) is used to define the dynamic threshold:

[0278] Threshold t = f(Season t , Variety t , GrowthStage t ,...) (9)

[0279] Where: Threshold t is the dynamic threshold at time t; f(⋅) is a nonlinear function that can be used to calculate the corresponding dynamic threshold based on specific factors, such as historical data statistics, expert experience, or machine learning methods (such as decision trees and random forests); Season t , Variety t , GrowthStage t They represent the season, planting variety and growth stage at time t respectively. The specific values of these factors can be provided by an external data source or estimated through the internal state of the plant growth analysis model. For example, the value of the growth stage factor can be the value obtained by processing the input data of the plant growth stage by the plant future growth prediction model.

[0280] 2. Data-driven preset threshold adjustment

[0281] The preset threshold function is recalculated periodically (e.g., weekly or monthly) to update the weights of seasonality, variety characteristics, and growth stage using newly collected data.

[0282] Exemplarily, the method for adjusting the preset threshold is shown in formula (10):

[0283] ΔThreshold t = Quantile α (X t-w:t ) - Quantile 1−α (X t−w:t ) (10)

[0284] Where: ΔThreshold t is the adjustment amount of the preset threshold at time t; X t−w:tis the observation data window from time t−w to time t; Quantile α (X) and Quantile 1−α (X) represents the αth quantile and the (1−α)th quantile of the dataset X, respectively. For example, α = 0.75 or 0.95 is selected, corresponding to the upper interquartile range (IQR) or twice the standard deviation.

[0285] 3. Early warning trigger and feedback mechanism

[0286] In practice, when actual observed data exceeds a preset threshold that can be dynamically adjusted, an early warning signal is triggered. Warning information should include the type of anomaly and the location of the anomaly. Furthermore, a user feedback mechanism should be established to collect user feedback on the accuracy of the warnings, which will be used to further optimize the dynamic threshold function and model update strategy.

[0287] In this embodiment, by dynamically adjusting the preset threshold according to factors such as season, variety, and growth stage, the abnormal threshold can be modulated to achieve adaptive early warning. Therefore, through adaptive early warning, it can be ensured that the plant growth analysis module can adapt to changes in plant growth conditions, identify and warn of growth anomalies in real time and accurately, and provide users with effective decision support.

[0288] In order to enable the gardening robot to simultaneously handle multiple tasks such as plant growth process reconstruction, plant future growth prediction, plant growth anomaly detection, etc., and provide comprehensive data analysis and prediction services, in one embodiment, such as Figure 15 As shown in Figure 2, the process analysis methods based on multimodal data include:

[0289] Step 1502, constructing an initial undirected graph; the initial undirected graph is used to represent an undirected graph in which potential connections exist between nodes and the initial connection weight is less than an initial threshold; the nodes include plant growth parameters and the number of environmental variables.

[0290] Step 1504 : Using the spatiotemporal feature vectors as inputs of the initial undirected graph, each spatiotemporal feature vector corresponds to a node of the initial undirected graph.

[0291] Step 1506 , embedding a memory unit during the node update process so that each node is attached with a corresponding memory state; the memory state is used to store historical information of the node.

[0292] Step 1508, in each message passing iteration, the node receives information from the node's neighbor nodes and interacts with the node's own memory state to update the node's memory state and hidden state.

[0293] Step 1510: Update the edge weights between nodes using the hidden states of the nodes and global context information to obtain an adaptive graph neural network.

[0294] Step 1512 , obtaining historical multi-dimensional data of the plant to be analyzed to obtain multimodal plant data; the historical multi-dimensional data at least includes plant physiological state data, plant morphological characteristic data, and environmental impact data.

[0295] Step 1514 , performing smoothing and normalization processing on the multimodal plant data to obtain processed multimodal plant data.

[0296] Step 1516 , performing time stamp synchronization processing on the processed multimodal plant data to obtain synchronized multimodal plant data; the timestamps of the synchronized multimodal plant data at the same time point are aligned.

[0297] Step 1518 : Associating each data in the synchronized multimodal plant data with the corresponding plant individual identifier to obtain traceable multimodal plant data.

[0298] Step 1520 , performing data integration on the traceable multimodal plant data to obtain a plant multimodal dataset.

[0299] Step 1522: Call the plant growth analysis model.

[0300] Step 1524: input the plant multimodal dataset into the spatiotemporal encoder of the plant growth analysis model, the spatiotemporal encoder including a first spatiotemporal convolutional layer, a second spatiotemporal convolutional layer, and a fully connected layer; input the time series dataset of the plant to be analyzed into the first spatiotemporal convolutional layer to obtain plant growth time features, and input the image dataset of the plant to be analyzed into the second spatiotemporal convolutional layer to obtain plant growth image features; the time series dataset and the image dataset are obtained by preprocessing the plant multimodal dataset.

[0301] Step 1526 , performing feature fusion on the plant growth time features and the plant growth image features through a fully connected layer to obtain a spatiotemporal feature vector.

[0302] In step 1528, the spatiotemporal feature vector is input into the spatiotemporal deconvolution layer of the spatiotemporal decoder. The spatiotemporal deconvolution layer increases the temporal and spatial resolution of the spatiotemporal feature vector to obtain plant growth history data. The plant growth history data includes reconstructed text data and reconstructed image data of the growth process.

[0303] Step 1530, obtain the number of future time steps.

[0304] In step 1532, the spatiotemporal feature vector is input into the adaptive graph neural network in the plant growth analysis model to obtain an adaptive graph structure. The spatiotemporal feature vector is obtained by processing the historical multimodal dataset through the spatiotemporal encoder of the plant growth analysis model. The adaptive graph structure is used to reflect the intrinsic relationship between different data modalities.

[0305] In step 1534, the adaptive graph structure and the number of future time steps are input into the spatiotemporal decoder in the plant growth analysis model to obtain plant future growth prediction data. The plant future growth prediction data includes key growth indicator prediction text data and growth status prediction image data.

[0306] Step 1536, the adaptive graph neural network also includes a memory unit layer, which extracts dependency information of the plant growth time characteristics through the memory unit layer to obtain long-term dependency information; the plant growth time characteristics are obtained by processing the time series data set of the plant growth environment of the plant to be analyzed through the first spatiotemporal convolution layer in the spatiotemporal encoder; the long-term dependency information is used to represent the historical growth trends and patterns of the plant to be analyzed.

[0307] In step 1538, the long-term dependency information is input into the adaptive graph neural network as an additional node feature to obtain an adaptive graph structure.

[0308] Step 1540: Acquire a single-frame plant image of the plant to be analyzed.

[0309] Step 1542 , using the real-time anomaly detection module in the plant growth analysis model, performs leaf feature extraction processing on the single-frame plant image to obtain leaf features corresponding to the plant to be analyzed.

[0310] Step 1544: determine whether there are any abnormalities in the leaves of the plant to be analyzed based on the leaf characteristics.

[0311] Step 1546: When there is an abnormality in the leaves of the plant to be analyzed, an immediate abnormality warning signal and an immediate abnormality detail report for the plant to be analyzed are output.

[0312] Step 1548 : When there is no abnormality in the leaves of the plant to be analyzed, obtain the plant time series data of the current time and the plant to be analyzed.

[0313] Step 1550: Outputting the plant growth prediction data corresponding to the current time of the plant to be analyzed through the spatiotemporal decoder in the plant growth analysis model.

[0314] Step 1552: Input the current plant growth prediction data and the plant time series data into the time series anomaly analysis module in the plant growth analysis model for comparison.

[0315] Step 1554: When the difference between the plant's current growth prediction data and the plant's time series data exceeds a preset threshold, it is confirmed that the plant to be analyzed has growth abnormalities, and a time series abnormality warning signal and a time series abnormality detail report are generated for the plant to be analyzed.

[0316] Step 1556 , when the difference between the current plant growth prediction data and the plant time series data does not exceed a preset threshold, confirm that the plant to be analyzed has no growth abnormality, and generate good plant growth status information for the plant to be analyzed.

[0317] Step 1558, based on the external data source or the internal state of the plant growth analysis model, estimate the key factor data affecting plant growth; the internal state of the plant growth analysis model is used to represent the state of the data related to the key factor data in the multimodal plant data after being processed by the plant growth analysis model.

[0318] Step 1560: Dynamically adjust the preset threshold based on the key factor data.

[0319] The above-mentioned process analysis method based on multimodal data has many advantages, including hierarchical and progressive learning, efficient utilization of common features, reduced model complexity, easy model maintenance and updating, saving training time and resources, and simplified deployment process. The details are as follows:

[0320] 1. Layered and progressive learning

[0321] In one embodiment, a plant growth reconstruction model serves as the foundational layer to learn the fundamental characteristics of plant growth. A plant future growth prediction model builds upon this foundation to predict future growth trends. A plant growth anomaly detection model further expands upon this foundation to detect anomalies. This layered, progressive learning strategy allows each model to further improve upon the previous one, leading to a deeper understanding of the complex process of plant growth.

[0322] 2. Efficiently utilize common features to reduce model complexity

[0323] Since the plant future growth prediction model and the plant growth anomaly detection model are built on the basis of the plant growth process reconstruction model, they can reuse the features extracted by the plant growth process reconstruction model. This reuse mechanism reduces the repeated feature extraction work and improves the computational efficiency.

[0324] By adding layers to an existing model to create a new model, you can avoid building a completely new complex model from scratch. This helps control the complexity of the model and reduce the risk of overfitting that may be caused by an overly complex model.

[0325] 3. Facilitate model maintenance and updates

[0326] When new data or new research findings are available, only the relevant model layers need to be updated rather than the entire model, which makes model maintenance and updates easier and more efficient.

[0327] 4. Save training time and resources

[0328] Since the underlying structure is shared between models, a lot of time and computing resources can be saved during training, which is especially important when the dataset is large or computing resources are limited.

[0329] 5. Simplify the deployment process

[0330] In actual applications, you can choose to enable or disable certain model layers as needed to provide customized solutions without redesigning or redeploying the entire system.

[0331] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0332] Based on the same inventive concept, embodiments of the present application also provide a multimodal data-based history analysis device for implementing the aforementioned multimodal data-based history analysis method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the multimodal data-based history analysis device provided below can be found in the aforementioned limitations of the multimodal data-based history analysis method and will not be further elaborated here.

[0333] In an exemplary embodiment, Figure 16 As shown, a process analysis device based on multimodal data is provided, including: an input module 1602, a model calling module 1604, a spatiotemporal convolution module 1606, a feature fusion module 1608 and an output module 1610, wherein:

[0334] The input module 1602 is used to obtain a historical multimodal dataset of the plant to be analyzed.

[0335] The model calling module 1604 is used to call the plant growth analysis model.

[0336] The spatiotemporal convolution module 1606 is used to input the plant multimodal dataset into the spatiotemporal encoder of the plant growth analysis model. The spatiotemporal encoder includes a first spatiotemporal convolution layer, a second spatiotemporal convolution layer, and a fully connected layer. The time series dataset of the plant to be analyzed is input into the first spatiotemporal convolution layer to obtain plant growth time characteristics, and the image dataset of the plant to be analyzed is input into the second spatiotemporal convolution layer to obtain plant growth image characteristics. The time series dataset and the image dataset are obtained by preprocessing the plant multimodal dataset.

[0337] A feature fusion module 1608 is configured to perform feature fusion on the plant growth time feature and the plant growth image feature through a fully connected layer to obtain a spatiotemporal feature vector;

[0338] The output module 1610 is used to input the spatiotemporal feature vector into the spatiotemporal decoder of the plant growth analysis model to obtain plant growth history data; the plant growth history data includes growth process reconstructed text data and growth process reconstructed image data.

[0339] In an exemplary embodiment, the output module 1610 includes:

[0340] The spatiotemporal deconvolution module is used to input the spatiotemporal feature vector into the spatiotemporal deconvolution layer of the spatiotemporal decoder, increase the temporal resolution and spatial resolution of the spatiotemporal feature vector through the spatiotemporal deconvolution layer, and obtain plant growth process data.

[0341] In an exemplary embodiment, the input module 1602 includes:

[0342] The acquisition module is used to obtain historical multi-dimensional data of the plant to be analyzed to obtain multimodal plant data; the historical multi-dimensional data at least includes plant physiological state data, plant morphological characteristic data and environmental impact data.

[0343] The preprocessing module is used to perform smoothing and standardization on the multimodal plant data to obtain processed multimodal plant data.

[0344] The timestamp alignment module is used to perform timestamp synchronization processing on the processed multimodal plant data to obtain synchronized multimodal plant data; the timestamps of the synchronized multimodal plant data at the same time point are aligned.

[0345] The association module is used to associate each data in the synchronized multimodal plant data with the corresponding plant individual identifier to obtain traceable multimodal plant data.

[0346] The data integration module integrates the traceable multimodal plant data to obtain a plant multimodal dataset.

[0347] In an exemplary embodiment, the multimodal data-based course analysis apparatus further includes:

[0348] The future time step input module is used to obtain the future time step.

[0349] The plant future growth prediction output module is used to output the plant future growth prediction data corresponding to the future time steps of the plant to be analyzed through the plant growth analysis model; the plant future growth prediction data includes key growth indicator prediction text data and growth status prediction image data.

[0350] In an exemplary embodiment, the plant future growth prediction output module includes:

[0351] An adaptive graph neural network module is used to input spatiotemporal feature vectors into the adaptive graph neural network in the plant growth analysis model to obtain an adaptive graph structure; the spatiotemporal feature vectors are obtained by processing the historical multimodal dataset through the spatiotemporal encoder of the plant growth analysis model; the adaptive graph structure is used to reflect the intrinsic correlation between different data modalities.

[0352] The data output module is used to input the adaptive graph structure and future time steps into the spatiotemporal decoder in the plant growth analysis model to obtain the plant future growth prediction data.

[0353] In an exemplary embodiment, the adaptive graph neural network further includes a memory unit layer, and the multimodal data-based history analysis device further includes:

[0354] The memory unit module is used to extract dependency information of plant growth time features through the memory unit layer to obtain long-term dependency information; the plant growth time features are obtained by processing the time series dataset of the plant growth environment of the plant to be analyzed through the first spatiotemporal convolutional layer in the spatiotemporal encoder; the long-term dependency information is used to represent the historical growth trends and patterns of the plant to be analyzed.

[0355] The adaptive graph structure output module inputs long-term dependency information as additional node features into the adaptive graph neural network to obtain an adaptive graph structure.

[0356] In an exemplary embodiment, a multimodal data-based history analysis device includes an adaptive graph neural network training module, and the adaptive graph neural network training module includes:

[0357] The initialization module is used to construct an initial undirected graph; the initial undirected graph is used to represent an undirected graph in which there are potential connections between nodes and the initial connection weight is less than an initial threshold; the nodes include plant growth parameters and the number of environmental variables.

[0358] The training input module is used to take the spatiotemporal feature vector as the input of the initial undirected graph, and each spatiotemporal feature vector corresponds to a node of the initial undirected graph.

[0359] A node update module is used to update the hidden state of a node based on its own features and neighbor node features in the initial undirected graph;

[0360] The edge weight update module is used to update the edge weights between nodes through the hidden states of the nodes and the global context information to obtain an adaptive graph neural network.

[0361] In an exemplary embodiment, the node update module of the training module of the adaptive graph neural network includes:

[0362] The training memory unit module is used to embed memory units during the node update process, so that each node is attached with a corresponding memory state; the memory state is used to store the historical information of the node.

[0363] The node memory update module is used to receive information from the node's neighbor nodes through the node in each message passing iteration, and interact with the node's own memory state to update the node's memory state and hidden state.

[0364] In an exemplary embodiment, the multimodal data-based course analysis apparatus further includes:

[0365] The real-time image acquisition module is used to obtain a single-frame plant image of the plant to be analyzed.

[0366] The feature extraction module is used to extract leaf features from single-frame plant images through the instant anomaly detection module in the plant growth analysis model to obtain leaf features corresponding to the plant to be analyzed.

[0367] The evaluation module is used to determine whether there are any abnormalities in the leaves of the plant to be analyzed based on the leaf characteristics.

[0368] The instant abnormality feedback module is used to output an instant abnormality warning signal and an instant abnormality detail report for the plant to be analyzed when there is an abnormality in the leaves of the plant to be analyzed.

[0369] In an exemplary embodiment, the multimodal data-based course analysis apparatus further includes:

[0370] The data acquisition module is used to acquire the plant time series data of the current time and the plant to be analyzed when there is no abnormality in the leaves of the plant to be analyzed.

[0371] The prediction data output module is used to output the plant current time growth prediction data corresponding to the current time of the plant to be analyzed through the spatiotemporal decoder in the plant growth analysis model.

[0372] The comparison module is used to compare the plant current time growth prediction data and the plant time series data input into the time series anomaly analysis module in the plant growth analysis model.

[0373] The abnormality feedback module is used to confirm that the plant to be analyzed has growth abnormalities when the difference between the plant's current time growth prediction data and the plant's time series data exceeds a preset threshold, and to generate a time series abnormality warning signal and a time series abnormality detail report for the plant to be analyzed.

[0374] The good feedback module is used to confirm that there is no growth abnormality in the plant to be analyzed when the difference between the plant current time growth prediction data and the plant time series data does not exceed a preset threshold, and generate good plant growth status information for the plant to be analyzed.

[0375] In an exemplary embodiment, the multimodal data-based course analysis apparatus further includes:

[0376] The key factor evaluation module is used to estimate the key factor data affecting plant growth based on external data sources or the internal state of the plant growth analysis model; the internal state of the plant growth analysis model is used to represent the state of data related to the key factor data in the multimodal plant data after being processed by the plant growth analysis model.

[0377] The threshold adjustment module is used to dynamically adjust the preset threshold based on key factor data.

[0378] Each module in the multimodal data-based process analysis device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor within a computer device in hardware form, or stored in memory within the computer device in software form, allowing the processor to call and execute the corresponding operations of each module. In a specific application, a gardening robot can be configured at the factory to perform three default tasks: plant growth process reconstruction, future plant growth prediction, and plant growth anomaly detection. This allows the gardening robot to simultaneously handle these tasks, providing comprehensive data analysis and prediction services. During use of the gardening robot, users can select different operating modes to display the results of corresponding models based on their specific needs and preferences. This design allows the gardening robot to flexibly adapt to the diverse needs of users, providing customized intelligent gardening management solutions.

[0379] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 17 As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication, and the wireless communication can be achieved via Wi-Fi, a mobile cellular network, NFC (near field communication), or other technologies. When executed by the processor, the computer program implements a process analysis method based on multimodal data.

[0380] Those skilled in the art will understand that Figure 17 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0381] In one embodiment, a gardening robot is further provided, comprising a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0382] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0383] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0384] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0385] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A process analysis method based on multimodal data, characterized in that: The method comprises: Obtain a historical multimodal dataset of the plants to be analyzed; Calling the plant growth analysis model; Inputting a plant multimodal dataset into a spatiotemporal encoder of the plant growth analysis model, the spatiotemporal encoder comprising a first spatiotemporal convolutional layer, a second spatiotemporal convolutional layer, and a fully connected layer, inputting a time series dataset of the plant to be analyzed into the first spatiotemporal convolutional layer to obtain plant growth time features, and inputting an image dataset of the plant to be analyzed into the second spatiotemporal convolutional layer to obtain plant growth image features; the time series dataset and the image dataset are obtained by preprocessing the plant multimodal dataset; the second spatiotemporal convolutional layer uses spatiotemporal convolution kernels for the time dimension and the space dimension; Performing feature fusion on the plant growth time feature and the plant growth image feature through the fully connected layer to obtain a spatiotemporal feature vector; Inputting the spatiotemporal feature vector into the spatiotemporal decoder of the plant growth analysis model to obtain plant growth history data includes: inputting the spatiotemporal feature vector into the spatiotemporal deconvolution layer of the spatiotemporal decoder, increasing the temporal resolution and spatial resolution of the spatiotemporal feature vector through the spatiotemporal deconvolution layer, and obtaining the plant growth history data; the plant growth history data includes growth process reconstructed text data and growth process reconstructed image data.

2. The method according to claim 1, characterized in that The step of obtaining a historical multimodal dataset of plants to be analyzed includes: Acquire historical multi-dimensional data of the plant to be analyzed to obtain multimodal plant data; the historical multi-dimensional data at least includes plant physiological state data, plant morphological characteristic data, and environmental impact data; performing smoothing and standardization processing on the multimodal plant data to obtain processed multimodal plant data; Performing time stamp synchronization processing on the processed multimodal plant data to obtain synchronized multimodal plant data; aligning time stamps of the synchronized multimodal plant data at the same time point; Associating each data in the synchronized multimodal plant data with a corresponding plant individual identifier to obtain traceable multimodal plant data; Data integration is performed on the traceable multimodal plant data to obtain the plant multimodal dataset.

3. The method according to claim 1, characterized in that The method further comprises: Get the number of future time steps; The plant growth analysis model also outputs plant future growth prediction data corresponding to the future time steps of the plant to be analyzed; the plant future growth prediction data includes key growth indicator prediction text data and growth status prediction image data.

4. The method according to claim 3, characterized in that The step of further outputting plant future growth prediction data corresponding to the future time steps of the plant to be analyzed through the plant growth analysis model includes: Inputting the spatiotemporal feature vectors into the adaptive graph neural network in the plant growth analysis model to obtain an adaptive graph structure; the spatiotemporal feature vectors are obtained by processing the historical multimodal dataset using the spatiotemporal encoder of the plant growth analysis model; the adaptive graph structure is used to reflect the inherent associations between different data modalities; The adaptive graph structure and the future time step number are input into a spatiotemporal decoder in the plant growth analysis model to obtain the future plant growth prediction data.

5. The method according to claim 4, characterized in that The adaptive graph neural network further includes a memory unit layer, and the method further includes: Dependency information extraction is performed on plant growth time features through the memory unit layer to obtain long-term dependency information; the plant growth time features are obtained by processing a time series dataset of the plant growth environment of the plant to be analyzed through the first spatiotemporal convolutional layer in the spatiotemporal encoder; the long-term dependency information is used to represent the historical growth trends and patterns of the plant to be analyzed; The long-term dependency information is input into the adaptive graph neural network as an additional node feature to obtain the adaptive graph structure.

6. The method according to claim 4, characterized in that The training process of the adaptive graph neural network is as follows: Constructing an initial undirected graph; the initial undirected graph is used to represent an undirected graph in which potential connections exist between nodes and the initial connection weight is less than an initial threshold; the nodes include plant growth parameters and environmental variable quantities; Using the spatiotemporal feature vector as input of the initial undirected graph, each of the spatiotemporal feature vectors corresponds to a node of the initial undirected graph; Based on the node characteristics and neighbor node characteristics of the node in the initial undirected graph, updating the hidden state of the node; The edge weights between the nodes are updated through the hidden states of the nodes and global context information to obtain the adaptive graph neural network.

7. The method according to claim 6, characterized in that The updating of the hidden state of the node based on the node's own features and neighbor node features in the initial undirected graph includes: Embedding a memory unit during the updating process of the node so that each node is attached with a corresponding memory state; the memory state is used to store historical information of the node; In each message passing iteration, information from neighbor nodes of the node is received by the node and interacts with the node's own memory state to update the node's memory state and hidden state.

8. The method according to claim 1, characterized in that The method further comprises: Acquire a single-frame plant image of the plant to be analyzed; Performing leaf feature extraction processing on the single-frame plant image through the instant anomaly detection module in the plant growth analysis model to obtain leaf features corresponding to the plant to be analyzed; determining whether there are abnormalities in the leaves of the plant to be analyzed according to the leaf characteristics; When there is an abnormality in the leaves of the plant to be analyzed, an immediate abnormality warning signal and an immediate abnormality detail report for the plant to be analyzed are output.

9. The method according to claim 8, characterized in that The method further comprises: When there is no abnormality in the leaves of the plant to be analyzed, obtaining plant time series data of the current time and the plant to be analyzed; Outputting the plant current time growth prediction data corresponding to the current time of the plant to be analyzed through the spatiotemporal decoder in the plant growth analysis model; Inputting the plant current time growth prediction data and the plant time series data into a time series anomaly analysis module in the plant growth analysis model for comparison; When the difference between the plant's current growth prediction data and the plant's time series data exceeds a preset threshold, confirming that the plant to be analyzed has growth abnormalities, and generating a time series abnormality warning signal and a time series abnormality details report for the plant to be analyzed; When the difference between the plant current time growth prediction data and the plant time series data does not exceed the preset threshold, it is confirmed that the plant to be analyzed has no growth abnormality, and good plant growth status information is generated for the plant to be analyzed.

10. The method according to claim 9, characterized in that The method further comprises: estimating key factor data affecting plant growth based on an external data source or an internal state of the plant growth analysis model; the internal state of the plant growth analysis model is used to represent a state of data related to the key factor data in the multimodal plant dataset after being processed by the plant growth analysis model; Based on the key factor data, the preset threshold is dynamically adjusted.

11. A machine history analysis device based on multimodal data, characterized in that: The device comprises: Input module, used to obtain historical multimodal datasets of plants to be analyzed; Model calling module, used to call plant growth analysis model; A spatiotemporal convolution module is configured to input a plant multimodal dataset into a spatiotemporal encoder of the plant growth analysis model, the spatiotemporal encoder comprising a first spatiotemporal convolution layer, a second spatiotemporal convolution layer, and a fully connected layer. The time series dataset of the plant to be analyzed is input into the first spatiotemporal convolution layer to obtain plant growth time features, and the image dataset of the plant to be analyzed is input into the second spatiotemporal convolution layer to obtain plant growth image features. The time series dataset and the image dataset are obtained by preprocessing the plant multimodal dataset. The second spatiotemporal convolution layer uses spatiotemporal convolution kernels for the time dimension and the space dimension. a feature fusion module, configured to perform feature fusion on the plant growth time feature and the plant growth image feature through the fully connected layer to obtain a spatiotemporal feature vector; The output module is used to input the spatiotemporal feature vector into the spatiotemporal decoder of the plant growth analysis model to obtain plant growth history data, including: inputting the spatiotemporal feature vector into the spatiotemporal deconvolution layer of the spatiotemporal decoder, increasing the temporal resolution and spatial resolution of the spatiotemporal feature vector through the spatiotemporal deconvolution layer, and obtaining the plant growth history data; the plant growth history data includes growth process reconstructed text data and growth process reconstructed image data.

Images