A training method and device of a growth milestone recognition model based on multi-modal feature fusion

Abstract

The application discloses a training method and device of a growth milestone recognition model based on multi-modal feature fusion, which is applied to the technical field of data processing. The application first collects a time sequence image sequence, maintenance operation records and environmental sensor data, generates visual features, time sequence features and semantic features, fuses the visual features, the time sequence features and the semantic features based on an EfficientNet-B4+ConvLSTM+YOLOv8 architecture and dynamically adapts weights, finally combines life specificity rules, a dynamic density self-adaptive algorithm and a time sequence NMS, optimizes inference effects and removes redundant frames. Through automatic image collection and intelligent analysis, a time-lapse video and a milestone note are generated, personalized reminders are provided based on environmental data and maintenance plans, and convenient state inquiries and operation records are realized through natural language interaction.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a training method and apparatus for a growth milestone recognition model based on multimodal feature fusion. Background Technology

[0002] Existing milestone identification solutions often suffer from the following drawbacks: Existing technologies rely solely on single-modal data, such as time-series images or environmental sensor data, lacking the collaborative acquisition and integration of multi-source data such as time-series image sequences, maintenance operation records, and environmental sensor data. This fails to fully reflect the complex influencing factors of organism growth. Furthermore, the data acquisition parameters are mostly fixed configurations, without dynamically adjusting the acquisition frequency and triggering conditions according to the growth / behavioral characteristics of different organisms such as plants, pets, and infants, resulting in the omission of key growth information or excessive redundant data.

[0003] Existing growth milestone recognition models lack a systematic design for extracting multimodal features such as visual, temporal, and semantic features. They fail to clearly define the quantification calculation methods for each feature sub-dimension, and feature fusion often employs a fixed weight strategy, failing to dynamically adapt weight ratios based on organism type, user preferences, and application scenarios. This results in insufficient effectiveness of feature fusion and limited recognition accuracy. During training, the model does not adequately consider the complexity of the dataset features and hardware computing resources to dynamically determine the training strategy, lacking targeted correction mechanisms for issues such as feature dimension mismatch, missing labels, and data distribution imbalance. Furthermore, the inference phase lacks organism-specific rules, dynamic density adaptive algorithms, and time-series NMS post-processing steps, leading to insufficient rationality in keyframe selection, excessive redundant frames, and poor adaptability to different organisms.

[0004] Existing solutions mostly focus on single identification functions, without linking practical functions such as pest and disease diagnosis, intelligent maintenance reminders, and QR code-based natural language interaction. They lack a complete chain design from data collection, growth milestone recognition model training, inference optimization to practical application, making it difficult to meet users' integrated needs for recording the growth of living organisms, maintenance management, and intelligent interaction. Summary of the Invention

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A training method for a growth milestone recognition model based on multimodal feature fusion includes: acquiring time-series image sequences, maintenance operation records, and environmental sensor data to generate visual features, time-series features, and semantic features; the growth milestone recognition model is based on a multimodal feature fusion architecture, learning feature maps through a multi-task learning strategy, and optimizing the parameters of the growth milestone recognition model according to the weighted fusion logic of visual, time-series, and semantic features. During the weighting process, the weights are dynamically adapted in combination with organism type, user preference, and application scenario. The multimodal feature fusion architecture consists of an EfficientNet-B4 backbone network + ConvLSTM time-series modeling + YOLOv8 detection head; the inference effect of the growth milestone recognition model is optimized by combining organism-specific rules and dynamic density adaptive algorithms, and a time-series NMS post-processing step is adopted to suppress redundant frames within the T time window before and after high-resolution keyframes, generating growth milestone recognition model inference optimization result information containing rule configuration, density parameters, suppression range, and optimization process.

[0006] A training apparatus for a growth milestone recognition model based on multimodal feature fusion, the apparatus being configured to perform any of the training methods for the growth milestone recognition model based on multimodal feature fusion described above.

[0007] Its beneficial effects are as follows: This invention provides a training method for a growth milestone recognition model based on multimodal feature fusion. This application integrates multi-source data from time-series images, maintenance records, and environmental sensors. The growth milestone recognition model extracts visual, temporal, and semantic features hierarchically, and fuses features through a dynamic weight adaptation mechanism. During training, indicators are verified in real time and dynamically adjusted. In the inference stage, it combines organism-specific rules, dynamic density adaptive algorithms, and time-series NMS to optimize keyframe selection and remove redundant frames. At the same time, it incorporates practical functions such as pest and disease diagnosis and QR code interaction to achieve end-to-end optimization. Through automatic image acquisition and intelligent analysis, it generates time-lapse videos and milestone notes; provides personalized reminders based on environmental data and maintenance plans; and enables convenient status queries and operation records through natural language interaction.

[0008] Multimodal data fusion and dynamic adaptation overcome the limitations of single-data sets, adjusting weights based on organism type, user preferences, and application scenarios to significantly improve the accuracy and adaptability of milestone recognition. Dynamic optimization of the training strategy addresses data quality issues through dimensional alignment and labeling, coupled with a multi-level adjustment mechanism to ensure the stability and convergence efficiency of the growth milestone recognition model training. A robust inference optimization mechanism effectively filters keyframes and removes redundancy through specific rules, density adaptive algorithms, and time-series NMS, improving the reasonableness of the results. Attached Figure Description

[0009] Figure 1A flowchart illustrating a training method for a growth milestone recognition model based on multimodal feature fusion, provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a training device for a growth milestone recognition model based on multimodal feature fusion, provided in an embodiment of the present invention. Detailed Implementation

[0010] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. Figure 1 This paper describes a training method for a growth milestone recognition model based on multimodal feature fusion according to an exemplary embodiment of the present application.

[0011] In this application embodiment, a training method for a growth milestone recognition model based on multimodal feature fusion is provided, such as... Figure 1 As shown: S101 acquires time-series image sequences, maintenance operation records, and environmental sensor data, and generates visual features, time-series features, and semantic features.

[0012] In one implementation, multimodal training data including time-series image sequences, maintenance operation records, and environmental sensor data, as well as coefficient parameters including type ID and weights, are acquired, and the multi-source data is processed into a standardized training dataset through a data preprocessing module. Adaptive acquisition mechanisms based on life form types acquire continuous image data: For plants, chlorophyll fluorescence detection is used to identify photosynthetic activity periods, automatically triggering shooting. Simultaneously, the acquisition frequency is dynamically adjusted based on new shoot density and leaf spread. For example, after watering or fertilizing, the acquisition frequency is increased in the short term to capture environmental response changes, such as increasing the acquisition frequency to once every 30 minutes within 12 hours after watering / fertilizing. For pets, motion sensors and sound detection trigger behavioral event shooting, and the timing of shooting is scheduled based on sleep-activity cycles and feeding times to capture scenes of play and social interaction. Multi-angle shooting is automatically switched when human-pet interaction is detected. For infants, developmental sensitive periods are predicted based on age, such as the critical period for rolling over at 3 months, with a focus on enhancing acquisition during critical periods. Close-up shots are also taken for events such as parent-child interaction and first vocalizations, while safety-related abnormal postures or location information are recorded simultaneously. Close-up shots are automatically taken when events such as parent-child interaction and first vocalizations are detected.

[0013] Record the entire life cycle maintenance behavior data of living organisms, including the execution time, operation method and related parameters of operations such as watering, fertilizing, pruning, and deworming. For example, in plant maintenance, records such as "watered 500ml on May 10, 2024" and "applied 10g of compound fertilizer on May 20, 2024"; in pet maintenance, information such as "dewormed on June 3, 2024" and "changed cat food on June 15, 2024"; in infant and toddler maintenance, content such as "added complementary food on July 5, 2024" and "vaccinated on July 12, 2024" are recorded to form a structured maintenance log.

[0014] Real-time environmental parameters, including temperature, humidity, light intensity, and soil moisture, are acquired through a sensor network, while third-party weather APIs are integrated to supplement environmental information. Data is continuously collected at fixed time intervals; for example, temperature, humidity, and light duration in the plant growth environment are recorded every hour; temperature and air quality in the pet activity environment are recorded every 30 minutes; and temperature, humidity, and noise levels in the environment of infants and young children are monitored in real time to ensure the continuity and integrity of environmental data.

[0015] Two types of core parameters are collected: first, type ID, which assigns a unique identifier to each life form, such as "succulent-001" for plants, "cat-002" for pets, and "baby-003" for infants, to distinguish data ownership; second, weight coefficients, which are preset with initial values ​​based on the life form type and application scenario. For example, in plant recognition, the weight of visual features is 0.4, the weight of temporal features is 0.3, and the weight of semantic features is 0.3; in infant recognition, the weight of semantic features is 0.5, the weight of visual features is 0.3, and the weight of temporal features is 0.2, providing a foundation for subsequent feature fusion.

[0016] The collected multimodal data is filtered and screened: blurry, occluded, and duplicate images are removed from the time-series image sequence; abnormal data in the maintenance operation records, such as records with incorrect format or missing parameters, are corrected; outliers and missing values ​​in the environmental sensor data are removed and supplemented with appropriate interpolation methods to ensure data accuracy.

[0017] Convert data from different sources into a standardized format: time-series image sequences are uniformly adjusted to the same resolution and color space (such as HSV space), and archived by naming in the order of timestamps; maintenance operation records are converted into structured text format, clearly specifying fields such as operation type, time, and parameters; environmental sensor data are converted into a unified numerical format, retaining a fixed number of decimal places, and organized and stored according to time series.

[0018] Basic feature preprocessing, such as noise reduction and grayscale correction, is performed on the image data. Numerical normalization is applied to the maintenance operation records and environmental sensor data to map parameters of different magnitudes to the same value range, eliminating the influence of units. Finally, all processed data are integrated to form a standardized training dataset containing time-series images, maintenance records, environmental data, type IDs, and weight coefficients, providing a unified input format for subsequent growth milestone recognition model training.

[0019] Optionally, in another embodiment based on the method described above in this application, the training platform reads the feature dimensions, data volume, and modality type information of the dataset, parses the standardized training dataset, and sorts the data according to modality importance.

[0020] In one implementation, feature dimension analysis is performed on the standardized training dataset to extract feature dimension information for each modality. For the multimodal data in the standardized training dataset, feature dimensions are analyzed separately for visual, temporal, and semantic modalities. Visual feature dimensions include three core dimensions: structural variation, color distribution variation, and motion / optical flow energy. The scores for each dimension and the overall calculation are as follows: Structural Change (S_structural): The difference between the current frame and the previous / reference frame is compared using SSIM (Structural Similarity Index) or MSE (Mean Squared Error). The larger the value, the more significant the structural change. The calculation formula is: S_structural=1-SSIM(F_t,F_{t-1}); F_t is the current frame image, which is the target frame to be analyzed in the time-series image sequence; F_{t-1} is the previous frame image (or reference frame), which serves as the reference frame for comparison with the current frame.

[0021] Color distribution change (S_color): Calculates the difference in the color histogram of HSV space (Bartholomew's distance, chi-square distance, etc.), which will be reflected in scenarios such as flower color change and leaf yellowing. The calculation formula is: S_color=cv2.compareHist(Hist_t,Hist_{t-1},cv2.HISTCMP_CHISQR); Motion / Optical Flow Energy (S_motion): Calculates the overall motion amplitude of a pixel using dense optical flow methods (such as Farneback), suitable for capturing rapid growth and sudden movements. The calculation formula is: S_motion = mean(||optical_flow||) over the frame; Visual comprehensive score: S_visual=α1*S_structural+α2*S_color+α3*S_motion, where α1, α2, and α3 are learnable weight coefficients, with a default value of 1 / 3, which can be dynamically adjusted according to the type of life form (e.g., the weight of α1 is increased to 0.4 in plant scenes).

[0022] The temporal feature dimensions include three core dimensions: acceleration of change, event proximity, and periodicity / pattern violation. The scores for each dimension and the overall calculation are as follows: Acceleration of change (S_acceleration): Calculates the second derivative of the change in visual features (such as outline area, subject height). The moment with large acceleration is the growth inflection point. The calculation formula is: S_acceleration=|d²(Height) / dt²| (for example, a scene where a plant suddenly grows taller). Event proximity (S_event): The closer the frame time point is to the maintenance event (watering, fertilizing, etc.), the higher the importance. The calculation formula is: S_event=1 / (1+|t_frame-t_event|), where t_frame is the current frame timestamp and t_event is the event timestamp; Periodicity / pattern violation (S_pattern): When a time series model with normal growth (such as an autoregressive model) is established, frames that deviate from the predicted value indicate anomalies or special phases. The calculation formula is: S_pattern=|observed_value-predicted_value| / std_dev, where std_dev is the standard deviation of the predicted value. Temporal comprehensive score: S_temporal=β1*S_acceleration+β2*S_event+β3*S_pattern, where β1, β2, and β3 are learnable weight coefficients, with a default value of 1 / 3. In the scientific report scenario, the weight of β2 is increased to 0.4.

[0023] The semantic feature dimension encompasses three core dimensions: milestone probability, emotional / aesthetic value, and narrative value. The scores for each dimension and the overall calculation are as follows: Milestone probability (S_milestone): Identifies key stages of a specific life form using a pre-trained model (such as EfficientNet, YOLO), and outputs a confidence score as a rating. The calculation formula is: S_milestone = Confidence_Score (model output). Among them, plant scenes detect "breaking soil", "new leaves", "buds", "flowering", "fruiting", etc.; infant and toddler scenes detect "rolling over", "crawling", "standing", "walking", etc.; and pet scenes detect "opening eyes", "walking", "playing", "specific behaviors".

[0024] Emotional / Aesthetic Value (S_aesthetic): The aesthetic rating developmental milestone recognition model is used to assess the "beauty" of the image. For example, flowers in the sunlight and smiling infants receive higher scores. The aesthetic rating value output by the neural network is used directly. Narrative Value (S_narrative): Analyze the semantic differences between the current frame and the selected keyframe sequence, evaluate its richness to the storyline (e.g., the frame of new shoots after the plant has withered has extremely high narrative value), and output a score through a narrative planner trained by reinforcement learning. Semantic comprehensive score: S_semantic=γ1*S_milestone+γ2*S_aesthetic+γ3*S_narrative, where γ1, γ2, and γ3 are learnable weight coefficients. In the emotional short film scene, the weights of γ2 and γ3 are both increased to 0.35, while γ1 is reduced to 0.3.

[0025] The sample size of the visual, temporal, and semantic modal data in the standardized training dataset was statistically analyzed. Visual modal samples were counted as single-frame images, with the total number of frames in all temporal image sequences included in the dataset. Temporal modal samples were counted as time-series data points, with the number of maintenance operation records and environmental sensor data collection times corresponding to each image frame, ensuring that each temporal data point was accurately associated with the timestamp of the visual sample. Semantic modal samples were counted as valid data with semantic labels, with the total number of samples labeled with milestone type (e.g., "new leaves" and "flowering" for plants, and "rolling over" and "crawling" for infants), emotional value scores, and narrative value assessment results. The distribution of valid samples under each semantic sub-dimension was also recorded, providing a basis for data balancing in the subsequent training of the growth milestone recognition model.

[0026] Modality identification is performed on the standardized training dataset to determine the visual, temporal, and semantic modal categories to which the data belongs. All data in the standardized training dataset are traversed, and their modality categories are determined based on data attributes and sources. Temporal image sequences from smart cameras, used to capture changes in image structure, color, and motion, are classified as visual modalities; data from maintenance operation logs, environmental sensor networks, and third-party weather APIs, used to reflect event correlations, trends, and cyclical patterns over time, are classified as temporal modalities; and data from growth milestone recognition models' semantic interpretation of image content (such as milestone recognition results), sentiment scoring, and narrative value analysis, used to reflect the meaning and value of the image, are classified as semantic modalities. Each type of data is labeled with its corresponding modality type to ensure accurate differentiation and retrieval during subsequent data processing and feature fusion.

[0027] A comprehensive evaluation of the parsed modal feature dimensions, data volume, and type information is conducted to generate a modal importance score. Combining the parsed modal feature dimensions, the number of statistical samples, and the identified modal types, a comprehensive evaluation system is constructed to generate the modal importance score. During the evaluation, the richness and discriminative power of the feature dimensions are considered first; modalities with more comprehensive dimensions and greater contribution to milestone recognition receive higher base scores. Next, the number of samples is considered; modalities with sufficient and evenly distributed samples receive higher scores, while those with scarce samples receive lower scores. Finally, weights are adjusted based on the type of living organism and the application scenario. For example, in plant recognition scenarios, the evaluation weights of visual modalities (reflecting changes in growth morphology) and temporal modalities (associating maintenance events with growth responses) are increased; in infant and toddler recognition scenarios, the evaluation weights of semantic modalities (capturing developmental milestones and emotional interactions) are increased; scientific report generation scenarios emphasize temporal and semantic modalities; and emotional short film generation scenarios emphasize semantic and visual modalities. The final importance score for each modality is obtained through weighted calculation.

[0028] The modal data is sorted based on modal importance scores to determine the priority of data loading and feature fusion. Based on the generated modal importance scores, visual, temporal, and semantic modal data are sorted in descending order. The sorting result directly determines the priority of data loading and feature fusion. During the loading phase, high-priority modal data is imported into the training platform first. For example, in a plant recognition scenario, if the visual modality score is the highest, followed by the temporal modality, and then the semantic modality score is the lowest, then the temporal image sequence data is loaded first, followed by maintenance operation records and environmental sensor data, and finally the semantically labeled data, ensuring that core data occupies computing resources first. During the feature fusion phase, high-priority modal features have higher weights in the weighted fusion. Multimodal features are weighted and fused according to the comprehensive key score formula S=w1S_visual+w2S_temporal+w3*S_semantic, where S_visual, S_temporal, and S_semantic are weighted averages. ntic represents the comprehensive scores for the visual, temporal, and semantic aspects mentioned above; w1, w2, and w3 are dynamic weights determined by the type of life form, user preferences, and application scenario: plant scenario (w1=0.4, w2=0.3, w3=0.3), infant scenario (w1=0.3, w2=0.2, w3=0.5), scientific report scenario (w1=0.2, w2=0.4, w3=0.4), and emotional short film scenario (w1=0.3, w2=0.2, w3=0.5). The higher the score of the modality, the larger the corresponding weight coefficient (w1, w2, w3) is, so as to maximize the contribution of the core modality to the training of the growth milestone recognition model and improve the accuracy of milestone recognition.

[0029] Optionally, in another embodiment based on the above method of this application, the training strategy is determined by the training platform and the growth milestone recognition model architecture through negotiation based on the feature complexity of the dataset and hardware computing resources, and the batch size and number of training iterations are determined according to the feature dimension and data volume. The dataset is then processed according to the target training strategy to form a training batch containing feature data and label information.

[0030] In one implementation, based on the dataset feature complexity evaluation results and hardware computing resources, the training platform and the growth milestone recognition model architecture negotiate to determine a multi-task learning training strategy. The batch size and number of training iterations are determined through feature dimension analysis and data volume statistics. First, a comprehensive evaluation of the feature complexity of the standardized training dataset is conducted, focusing on the feature dimension richness, data distribution complexity, and intermodal correlation of the visual, temporal, and semantic modalities. Simultaneously, hardware computing resource information is acquired, including GPU memory size, processing speed, CPU processing power, and memory capacity. The training platform and the growth milestone recognition model architecture (EfficientNet-B4 backbone network + ConvLSTM temporal modeling + YOLOv8 detector head) negotiate to determine the adoption of a multi-task learning training strategy. This strategy enables efficient sharing of multimodal features, reduces the risk of overfitting, and improves the generalization ability of the growth milestone recognition model. Combining the previously analyzed modal feature dimension information (such as visual features including structure, color, and motion sub-dimensions) and data volume statistics (the number and distribution of samples in each modality), and taking into account the hardware computing power capacity, the batch size and training iteration number are reasonably determined to ensure that hardware resources are not exceeded during training, while ensuring that the growth milestone recognition model can fully learn the data features.

[0031] A data partitioning mechanism was employed to split the standardized training dataset, extracting feature data according to visual, temporal, and semantic modal types, and associating them with corresponding milestone label information to generate initial training batches. A reasonable data partitioning mechanism was used to split the standardized training dataset according to a preset ratio (e.g., an 8:2 ratio for training and validation sets). During the splitting process, visual, temporal, and semantic feature data were extracted according to modal type: visual feature data was extracted from temporal image sequences, covering structural changes, color distribution changes, and motion / optical flow energy-related features; temporal feature data was extracted from maintenance operation records and environmental sensor data, including acceleration changes, event proximity, and periodicity / pattern violation-related features; semantic feature data was extracted from labeled milestone information, sentiment scores, and narrative value assessment results, including milestone probability, sentiment / aesthetic value, and narrative value-related features. The extracted modal feature data were then associated with corresponding milestone label information, such as "new leaves" and "flowering" labels for plants, and "rolling over" and "crawling" labels for infants, forming initial training batches to ensure that each training sample contains complete feature data and clear label information.

[0032] A comprehensive validity check is performed on the initial training batch: To address the issue of inconsistent dimensionality among different modal feature data, a dimension alignment service is initiated. Through feature mapping, dimension expansion, or compression, all modal feature data is adjusted to a unified dimension, ensuring the feasibility of feature fusion. For samples with missing labels, a labeling mechanism is triggered, combining contextual features, similar sample labels, and domain knowledge to supplement missing-label samples with appropriate milestone labels. To address the data distribution imbalance caused by an excessive or insufficient number of samples in certain milestone types, a resampling process is executed. Categories with an excessive number of samples are downsampled, while categories with an insufficient number of samples are upsampled, balancing the sample ratio across categories. Based on the above data correction operations, an optimized training batch scheme is generated, including data correction strategies such as dimension alignment methods, labeling rules, resampling ratios, and a sample balancing scheme.

[0033] Following the optimized training batch scheme, data from each modality is integrated and processed. A parallel loading mechanism is used to simultaneously import data of the same modality and with the same weights into the training platform. The Celery + Redis asynchronous task processing mechanism is utilized to improve data loading efficiency and shorten training preparation time. During data loading, training batch index information is generated synchronously, clearly defining the position and modality category of each sample in the dataset, facilitating rapid access and management during training. Real-time feedback on data loading progress allows operators to clearly understand the loading status. Once all data is loaded, the final training batch is formed. This batch contains corrected and balanced visual, temporal, and semantic feature data, along with corresponding milestone label information, which can be directly used for training the growth milestone recognition model.

[0034] Optionally, in another embodiment based on the method described above in this application, the training platform transmits training data according to batch size and number of iterations, and data of the same modality and with the same weight are imported through a parallel loading mechanism.

[0035] In one implementation, the training platform strictly adheres to a predetermined batch size and training iteration count, establishing an orderly data transmission plan. Before transmission, the platform categorizes and organizes the data in the final training batch, ensuring that each batch contains complete visual, temporal, and semantic multimodal feature data and corresponding milestone label information, with the total data volume perfectly matching the batch size. During transmission, the platform gradually pushes data according to the training iteration count. After each batch of data transmission and growth milestone recognition model training is completed, the next batch transmission is initiated, ensuring the training process proceeds sequentially and avoiding data congestion or transmission omissions.

[0036] Before data transmission, the training platform filters and groups the data, grouping data belonging to the same modality within the visual, temporal, and semantic domains with identical weight coefficients. For example, in the visual modality, structural change and color distribution change related features with the same weight are grouped together; in the temporal modality, event proximity and acceleration of change related features with consistent weights are grouped together; and in the semantic modality, milestone probability and sentiment / aesthetic value related features with consistent weights are grouped separately. This ensures that the modal attributes and weight configurations of each group of data are completely consistent, providing a data foundation for parallel loading.

[0037] Leveraging the Celery+Redis asynchronous task processing mechanism, the training platform initiates a parallel loading process, simultaneously importing grouped data of the same modality and weight. Different groups of data occupy independent loading channels, ensuring no interference and significantly improving data import efficiency while shortening training preparation time. During loading, the system monitors the data transmission progress of each channel in real time, generating loading progress feedback to ensure operators are aware of the data import status. Once loading is complete, the data is directly sent to the growth milestone recognition model training module (EfficientNet-B4 backbone network + ConvLSTM temporal modeling + YOLOv8 detector head), providing timely and complete data support for multimodal feature extraction and fusion in the growth milestone recognition model.

[0038] Optionally, in another embodiment based on the method described above in this application, the loss value and accuracy index are checked in real time during the training process of the growth milestone recognition model. If the index is abnormal, an adjustment mechanism is triggered until the index is stable.

[0039] In one implementation, based on the real-time requirements of training the growth milestone recognition model and the indicator monitoring objectives, loss values ​​and accuracy indicators are collected and standardized in real time to generate indicator monitoring data containing numerical magnitude, trend of change, and threshold range. Combining the real-time requirements of the growth milestone recognition model training and the indicator monitoring objectives, loss values ​​and accuracy indicators are continuously collected during the training process. The loss value reflects the degree of deviation between the prediction results of the growth milestone recognition model and the true labels, while the accuracy indicator reflects the accuracy of the growth milestone recognition model in identifying milestones. The collected raw data is standardized to eliminate dimensional differences caused by different batches and modalities of data, and to unify the data format and value range. The final generated indicator monitoring data must contain three core pieces of information: numerical magnitude (real-time loss value and real-time accuracy values), trend of change (whether the loss value continues to decrease, whether the accuracy continues to increase, and the rate of change), and threshold range (a reasonable range of loss values ​​and an accuracy threshold preset based on the training objectives of the growth milestone recognition model), providing a clear data foundation for subsequent anomaly detection.

[0040] Based on the convergence characteristics of the growth milestone recognition model, judgment rules for monitoring data are designed, clarifying the anomaly judgment thresholds for stagnant loss value decline and excessive accuracy fluctuations, and generating anomaly recognition standards for indicators. Based on the convergence characteristics of the growth milestone recognition model, the judgment dimensions and thresholds for indicator anomalies are clarified, forming anomaly recognition standards for indicators. For loss values, if the decrease in loss value is lower than a preset threshold over multiple consecutive training iterations, it is judged as stagnant loss value decline. For accuracy indicators, if the accuracy fluctuation exceeds a set upper limit during training, or if the accuracy reaches a certain value and then stops improving or even decreases, it is judged as excessive accuracy fluctuation. By quantifying the judgment thresholds for these anomaly scenarios, the boundary between normal and abnormal training states is clearly defined, ensuring that anomalies can be accurately identified.

[0041] To address the challenges of multi-task learning training strategies and parameter optimization, a multi-level adjustment mechanism is implemented, encompassing dynamic learning rate adjustment, batch size adaptation, and weight coefficient calibration. This ensures rapid response and optimization when metrics become abnormal. Specifically, a three-level adjustment mechanism is established: Level 1: Dynamic learning rate adjustment. When the loss value stagnates, the learning rate is appropriately reduced to fine-tune the growth milestone recognition model parameters. When accuracy improvement is slow, the learning rate can be increased in stages to accelerate convergence. Level 2: Batch size adaptation. Based on hardware computing power and data distribution, if training instability leads to accuracy fluctuations, the batch size can be adjusted to balance training efficiency and stability. Level 3: Weight coefficient calibration. Based on the importance scores of visual, temporal, and semantic modalities and the training feedback of the growth milestone recognition model, the fusion weights of each modality feature are calibrated. If a certain modality feature's contribution is insufficient, resulting in low accuracy, its weight ratio can be appropriately increased.

[0042] The system integrates and executes indicator monitoring data, anomaly identification criteria, and multi-level adjustment mechanisms to continuously iterate and optimize the parameters of the growth milestone recognition model. This generates training process control information including monitoring results, adjustment actions, and indicator convergence status, until the loss value and accuracy indicators stabilize within a reasonable range. The indicator monitoring data, anomaly identification criteria, and multi-level adjustment mechanisms are integrated into the closed-loop optimization process of the growth milestone recognition model training. During training, monitoring data and anomaly identification criteria are compared in real time. Once an indicator anomaly is detected, the corresponding adjustment mechanism is immediately triggered: if the loss value is determined to have stalled, the learning rate is dynamically adjusted; if the accuracy fluctuation exceeds the limit, the batch size is adjusted or the weight coefficients are calibrated based on the situation. The "collection-judgment-adjustment" process is continuously iterated. After each adjustment, indicator data is re-collected and the adjustment effect is evaluated, generating training process control information including monitoring results (anomaly type, anomaly duration), adjustment actions (adjusted parameter type, adjustment magnitude), and indicator convergence status (whether it gradually stabilizes). The parameters of the growth milestone recognition model are repeatedly iterated and optimized until the loss value stabilizes within a preset reasonable range and the accuracy index meets the target requirements and remains stable, thus completing the convergence optimization of the growth milestone recognition model training.

[0043] S102, the growth milestone recognition model is based on a multimodal feature fusion architecture. It learns feature mapping through a multi-task learning strategy and optimizes the parameters of the growth milestone recognition model according to the weighted fusion logic of visual, temporal and semantic features. During the weighting process, the weights are dynamically adapted by combining the life type, user preferences and application scenarios.

[0044] In one implementation, based on the requirements of multimodal feature fusion and the goal of milestone recognition, a fusion architecture of EfficientNet-B4 backbone network + ConvLSTM temporal modeling + YOLOv8 detection head is constructed to perform hierarchical extraction and feature mapping learning of visual, temporal, and semantic features. Based on the requirements of multimodal feature fusion and the goal of accurate milestone recognition, a three-level fusion architecture of "EfficientNet-B4 backbone network + ConvLSTM temporal modeling + YOLOv8 detection head" is constructed. The architecture and connections are clearly defined: First, the EfficientNet-B4 backbone network serves as the core for feature extraction. It utilizes a composite scaling strategy to uniformly adjust depth, width, and resolution, achieving strong feature extraction capabilities with fewer parameters (8 times less than ResNet-50). This adapts to scenarios such as fine-grained features in plant images, extracting visual features (structure, color, motion dimension), temporal features (acceleration of change, event proximity, etc.), and semantic features (milestone probability, sentiment value, etc.) layer by layer. Next, the extracted features are input into the ConvLSTM temporal modeling network. This network combines the spatial feature preservation capabilities of convolution with the temporal modeling advantages of LSTM, along with a multi-head self-attention mechanism to capture long-distance dependencies, adapting to the learning needs of temporal features such as slow changes in plant growth. Finally, the YOLOv8 detection head connects, combining object detection (locating new organs) and classification (identifying milestone types), offering high speed and accuracy, and supporting real-time detection. The entire architecture achieves feature mapping learning through the process of "feature extraction → temporal modeling → detection and classification", ensuring that the three types of modal features are fully explored and transformed into effective information that can be used by the growth milestone recognition model.

[0045] This paper designs a feature fusion logic based on a multi-task learning strategy, clarifying the extraction dimensions and fusion ratios of visual features, temporal features, and semantic features, and generating feature fusion rules. The feature fusion logic is designed based on a multi-task learning strategy, generating explicit feature fusion rules. The extraction dimensions for each modality are defined as follows: visual feature extraction includes three dimensions: structural changes (SSIM / MSE quantization), color distribution changes (HSV spatial histogram differences), and motion / optical flow energy (dense optical flow statistics); temporal feature extraction includes three dimensions: acceleration of change (second derivative of visual features), event proximity (time distance from maintenance events), and periodicity / pattern violation (deviation from time series model predictions); semantic feature extraction includes three dimensions: milestone probability (model confidence), emotional / aesthetic value (neural network scoring), and narrative value (semantic differences and narrative template fit). Simultaneously, the fusion ratio rules are defined, based on the comprehensive key scoring formula S=w1S_visual+w2S_temporal+w3*S_semantic, setting the basic ratio of each modality feature during fusion to ensure the orderliness and rationality of feature fusion, providing a benchmark for subsequent dynamic adjustments.

[0046] A dynamic weight adjustment mechanism is established to adapt to different life forms, user preferences, and application scenarios. For plants, the emphasis is on visual and temporal characteristics; for infants and toddlers, on semantic and emotional characteristics; for scientific reports, on temporal and semantic characteristics; and for emotional short films, on semantic and visual characteristics. Specifically, for plant life forms: the emphasis is on visual and temporal characteristics, as changes in growth are primarily reflected through morphological and temporal changes; for infants and toddlers: the emphasis is on semantic and emotional characteristics, focusing on developmental milestones and moments of emotional interaction. For application scenarios: the emphasis is on temporal and semantic characteristics in scientific reports, requiring accurate presentation of growth patterns and the significance of key milestones; the emphasis is on semantic and visual characteristics in emotional short films, focusing on beautiful moments and visual presentation. Regarding user preferences, if users prefer "the process of change," the weight of temporal and visual characteristics is increased; if they prefer "beautiful moments," the weight of semantic and visual characteristics is increased, ensuring that the weight adjustments align with actual usage needs.

[0047] The established fusion architecture, designed feature fusion rules, and dynamic weighting mechanism are integrated into a multi-task learning training process. The training steps are as follows: First, standardized training batch data is input into the fusion architecture, and three modal features are extracted according to preset dimensions. Next, preliminary fusion is performed according to the feature fusion rules. Then, the dynamic weighting mechanism is used to adapt to the current life form type, user preferences, and application scenario, adjusting the weight ratio of each modality. Subsequently, joint training is conducted through a multi-task learning strategy to improve feature sharing capabilities, reduce overfitting, and optimize the parameters of each layer of the architecture (feature extraction parameters of EfficientNet-B4, temporal modeling parameters of ConvLSTM, and detection and classification parameters of YOLOv8). During training, the growth milestone recognition model outputs training result information including feature extraction results (specific values ​​for each modal dimension), weight configuration (dynamically adjusted w1, w2, and w3 values), and fusion output (final feature vector). Through continuous iterative optimization, efficient fusion of multimodal features is achieved, ensuring the accuracy of milestone recognition.

[0048] S103 combines organism-specific rules and dynamic density adaptive algorithms to optimize the inference performance of the growth milestone recognition model. At the same time, it adopts a time-series NMS post-processing step to suppress redundant frames within the T time window before and after high-scoring keyframes.

[0049] In one implementation, based on the inference optimization requirements of the growth milestone recognition model, a pre-trained multimodal fusion growth milestone recognition model (EfficientNet-B4 backbone network + ConvLSTM temporal modeling + YOLOv8 detection head architecture) is obtained from the training environment. This growth milestone recognition model has been optimized through multi-task learning and possesses the ability to weightedly fuse visual, temporal, and semantic features and dynamically adapt weights. Simultaneously, organism-specific rules (new organ development detection in plants, key flowering and fruiting periods, developmental milestone detection in infants, safety alarm-related rules, etc.) and parameters related to the dynamic density adaptive algorithm (initial values ​​of learnable parameters α, β, and γ, calculation benchmarks for content change rate, information entropy, and semantic importance, etc.) are collected to ensure that all core resources required for inference optimization are ready, providing data and rule support for subsequent optimization steps.

[0050] A comprehensive validity verification was performed on the initial inference results of the multimodal fusion growth milestone recognition model. The core verification dimensions included two aspects: first, the milestone recognition confidence level, determining whether the confidence score corresponding to the milestone type output by the growth milestone recognition model (such as "new leaf" in plants, "crawling" in infants) reached the preset qualified threshold, and filtering out low-confidence misidentification results; second, the rationality of keyframe selection, checking whether the selected keyframes accurately correspond to milestone events, and whether there were any omissions of key growth nodes or misselection of irrelevant frames. Based on the verification results, an inference result verification report was generated, clarifying issues such as insufficient confidence and keyframe selection bias in the initial inference, providing targeted directions for subsequent rule application and algorithm optimization.

[0051] To meet the adaptation needs of different life forms, specific rule application standards are formulated. For plants, the standards define the detection of new organ development (structural change threshold), the determination of key flowering and fruiting periods (color and texture feature standards), and color anomalies based on visual features such as the proportion of yellowing leaf area and texture changes such as spot density. These are combined with environmental responses in temporal features, such as trends under high humidity, to identify early signs of diseases and output disease diagnosis results, including disease type (e.g., powdery mildew, aphid infestation), confidence score, and severity level. Targeted prevention and control suggestions (e.g., pesticide type, dosage, environmental adjustment plan) are generated by linking to the maintenance plan library and updated to the corresponding plant_id maintenance record. For infants and young children, the standards define the triggering conditions for developmental milestone detection (feature parameters corresponding to each milestone), enhanced social interaction (semantic feature judgment standards), learning moment recording (behavioral feature extraction rules), and safety alarms (health indicator safety thresholds). Rule application parameters are generated based on the application standards, including judgment thresholds for various features and the priority order of rule triggering, ensuring that specific rules can accurately adapt to the reasoning optimization needs of different life forms.

[0052] Dynamically adjust keyframe density based on content richness, and select density function on a frame-by-frame basis. Weight coefficients for content change rate, information entropy, and semantic importance are set to generate an adaptive density configuration. Here, C(t) represents the content change rate (quantified through differences in inter-frame structure, color, and motion features), I(t) represents the information entropy (reflecting the richness of information within a frame), and S(t) represents semantic importance (calculated based on a comprehensive analysis of milestone probability, emotional value, and narrative value). α, β, and γ are learnable parameters used to adjust the weights of these three dimensions on keyframe density. The values ​​of α, β, and γ are optimized during the training of the growth milestone recognition model to generate the adaptive density configuration. This allows the keyframe density to be dynamically adjusted according to content richness—increasing the number of keyframes during periods of drastic content change, high information entropy, and strong semantic importance, and decreasing them conversely, ensuring that keyframes comprehensively cover important nodes without redundancy.

[0053] To address the issue of keyframe temporal clustering, temporal NMS processing rules are established, clarifying the suppression range and score adjustment strategy for the T-time window before and after high-scoring keyframes to avoid retaining redundant frames. To address the potential for excessive temporal clustering of keyframes in the initial inference results, temporal NMS processing rules are also defined. The core optimization strategy is as follows: First, high-scoring keyframes (keyframes with high confidence rankings) are identified, and a T-time window (e.g., a range of 3 seconds or 5 frames) before and after them is defined as the suppression range. Within this time window, the scores of other candidate keyframes are adjusted, lowering their priority to prevent multiple frames with similar content from being selected as keyframes simultaneously. This rule effectively removes temporally clustered redundant frames, ensuring a uniform distribution of keyframes in the temporal dimension, preserving core growth nodes while avoiding duplicate recording.

[0054] By integrating rule application parameters, density adaptive configuration, and time-series NMS rules, an operational specification for inference optimization of the growth milestone recognition model was formulated, clarifying the rule calling order and parameter adjustment process. The previously generated rule application parameters (judgment thresholds and trigger priorities for organism-specific rules), density adaptive configuration (α, β, γ values, and C(t), I(t), S(t) calculation standards), and time-series NMS rules (T time window size and score adjustment strategy) were integrated to formulate an operational specification for inference optimization of the growth milestone recognition model. The rule calling order was clarified: first, organism-specific rules were applied to screen valid milestones and corresponding keyframes; then, the keyframe density was adjusted using a dynamic density adaptive algorithm; and finally, time-series NMS rules were used to remove redundant frames. Simultaneously, the parameter adjustment process was clarified, stipulating that when the inference result verification report shows a certain type of problem (such as a large number of low-confidence milestones), the judgment threshold or algorithm parameters of the corresponding rule should be adjusted first to ensure that the optimization operation proceeds in an orderly and efficient manner.

[0055] The inference result verification report (initial inference problem list), rule application parameters (specific rule thresholds and priorities), density adaptive configuration (function parameters and calculation standards), temporal NMS rules (suppression range and score strategy), and inference optimization operation specifications (call order and adjustment process) are comprehensively integrated to generate complete inference optimization result information for the growth milestone recognition model. This information includes rule configuration details, density parameter values, redundant frame suppression range, and a complete optimization process. It can be directly used to guide the optimization of the inference process of the multimodal fusion growth milestone recognition model, ultimately achieving improved milestone recognition accuracy, reasonable key frame distribution, and effective removal of redundant frames.

[0056] The growth milestone recognition model assigns a unique `plant_id` to each living organism, generates an associated QR code, and creates a session and assigns a `session_token` (valid for 30 minutes) upon user scanning. Session context and dialogue history are stored in Redis. It supports four core intents: status query, maintenance reminders, problem diagnosis, and operation logs. Based on an NLP engine, it parses user natural language and generates responses using multimodal data. A memory network maintains the dialogue context, associating the `plant_id` with corresponding maintenance records and environmental data to achieve continuous question-and-answer functionality (e.g., when a user asks "What should I do if my plant leaves are turning yellow?", the system automatically retrieves recent images and environmental data, outputting diagnostic results and suggestions).

[0057] Optionally, in another embodiment based on the method described above in this application, the method further includes: Obtain information about the object to be monitored; The growth milestone recognition model is used to process the information of the object to be monitored, and generate milestone recognition results, key frame screening results and supporting auxiliary information for the object to be monitored. The supporting auxiliary information includes disease diagnosis related data, structured growth record data and conversation interaction data.

[0058] In one implementation, the information of the object to be monitored is collected from all dimensions of the living organism to ensure the integrity and relevance of the information. Specifically, it includes three types of core information: basic identification information includes the type of the object to be monitored (plant, pet, infant) and unique identifier (plant_id, etc.), which is used to associate with the corresponding digital archive (such as the species and maintenance history in the plant archive).

[0059] The multimodal sensing data includes: real-time time-series image sequences collected by smart cameras (adaptively collected according to the type of living organism, such as photosynthetic activity of plants and close-ups of sensitive periods of infants); environmental data (temperature, humidity, soil moisture, etc.) obtained through sensor networks; and maintenance operation records (time and parameters of operations such as watering, fertilizing, and pest control) manually entered by the user or automatically recorded by the system.

[0060] The interactive information is as follows: supplementary information uploaded by the user when initiating a query through the QR code session module (such as images of suspected lesions, questions described in natural language), and session history context data.

[0061] The growth milestone recognition model (EfficientNet-B4 backbone network + ConvLSTM temporal modeling + YOLOv8 detection head architecture) performs multimodal fusion processing on the above information and outputs three core results: 1. Milestone recognition results: The model accurately identifies the growth milestones of the object under monitoring by extracting visual, temporal, and semantic features in a hierarchical manner and combining them with a dynamic weight adaptation mechanism. For plant scenarios: Identify key stages such as "breaking through soil", "new leaves", "flower buds", "flowering", and "fruiting", and output milestone type, corresponding timestamp, and identification confidence score (confidence value output by the model).

[0062] For infants and toddlers: identify developmental milestones such as "rolling over", "sitting independently", "crawling", "standing", and "walking", and output milestone achievement status and confidence level in conjunction with age information.

[0063] For pet scenarios: Identify specific behavioral milestones such as "eyes open", "walking", and "playing", and output the identification results by combining physiological cycle data.

[0064] 2. Keyframe screening results, after multimodal feature scoring, organism-specific rule optimization, dynamic density adaptive adjustment, and time-series NMS post-processing, selected non-redundant keyframe sequences. The keyframe storage path (associated with cloud storage image / video resources) and timestamp (precisely aligned with maintenance operation records and environmental data) are included. Auxiliary information includes the comprehensive criticality score for each frame (S = w1S_visual + w2S_temporal + w3*S_semantic), density adaptive configuration parameters (α, β, γ values), and the frame distribution after time-series NMS processing (to avoid temporal clustering redundancy).

[0065] 3. Supporting auxiliary information: For disease diagnosis-related data, if the model detects abnormal objects (such as yellowing of plant leaves or abnormal pet behavior), it combines visual features (abnormal color, texture changes) and environmental data (high humidity, insufficient light) to output the disease type, diagnostic confidence level, severity level, and corresponding prevention and control suggestions (such as agent type, application dosage, and environmental adjustment plan), and updates the early warning records in the maintenance plan library.

[0066] For structured growth record data, the system automatically organizes milestone identification results, keyframe information, maintenance operation records, and environmental data to generate a standardized growth log, including a milestone achievement timeline, growth / development indicator change trends (such as the increase in the number of plant leaves, and milestones related to infant height and weight), and an analysis of the impact of environmental factors on growth. This log can be directly used to generate illustrated story cards and interactive timelines.

[0067] For conversational interaction data, it adapts to the natural language interaction requirements of the QR code conversation module, outputting structured information that can be directly converted into natural language responses, including a summary of the current state of the object (e.g., "The plant is at the 'new leaf' milestone, and it has been 3 days since the last watering"), reminders of incomplete maintenance (e.g., "Fertilizer needs to be applied next week"), and targeted answers to user questions (e.g., responding to the question "The leaves are turning yellow" based on the disease diagnosis results). At the same time, it updates the conversation context to the memory network to support multi-turn dialogue.

[0068] like Figure 2 As shown, a training device for a growth milestone recognition model based on multimodal feature fusion includes: The data acquisition and preprocessing module 201 is used to acquire time-series image sequences, maintenance operation records, and environmental sensor data, and generate visual features, time-series features, and semantic features. The feature fusion training module 202 is used to build a fusion architecture of EfficientNet-B4 backbone network + ConvLSTM temporal modeling + YOLOv8 detection head. It extracts visual, temporal and semantic features in layers through a multi-task learning strategy, optimizes the growth milestone recognition model parameters according to weighted fusion logic, and dynamically adapts weights based on life type, user preference and application scenario. The inference optimization module 203 is used to optimize the inference effect of the growth milestone recognition model by combining organism-specific rules and dynamic density adaptive algorithm. At the same time, it adopts the time-series NMS post-processing step to suppress redundant frames within the T time window before and after the high-scoring key frame, and generates inference optimization result information of growth milestone recognition model including rule configuration, density parameters, suppression range and optimization process.

[0069] A computing device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute any training method of a growth milestone recognition model based on multimodal feature fusion.

[0070] The methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.

[0071] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0072] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0073] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application.

Claims

1. A training method for a growth milestone recognition model based on multimodal feature fusion, characterized in that, include: Acquire time-series image sequences, maintenance operation records, and environmental sensor data to generate visual features, time-series features, and semantic features; The growth milestone recognition model is based on a multimodal feature fusion architecture. It learns feature mapping through a multi-task learning strategy and optimizes the parameters of the growth milestone recognition model according to the weighted fusion logic of visual, temporal and semantic features. During the weighting process, the weights are dynamically adapted by combining the life type, user preferences and application scenarios. The multimodal feature fusion architecture consists of EfficientNet-B4 backbone network + ConvLSTM temporal modeling + YOLOv8 detection head. The reasoning performance of the growth milestone recognition model is optimized by combining organism-specific rules and dynamic density adaptive algorithms. At the same time, a time-series NMS post-processing step is adopted to suppress redundant frames within the T time window before and after the high-scoring key frame, generating growth milestone recognition model reasoning optimization result information that includes rule configuration, density parameters, suppression range and optimization process.

2. The training method for the growth milestone recognition model based on multimodal feature fusion according to claim 1, characterized in that, Acquire time-series image sequences, maintenance operation records, and environmental sensor data to generate visual features, temporal features, and semantic features, including: The images are acquired through a multimodal adaptive acquisition system. For plants, the acquisition frequency is dynamically adjusted according to the photosynthetic activity period, growth stage, and environmental response. For pets, the acquisition is triggered according to behavioral patterns, physiological cycles, and social interactions. For infants and young children, the acquisition is enhanced according to their developmental sensitive periods, safety monitoring, and parent-child interaction, forming a time-series image sequence. Record the entire life cycle maintenance behavior of living organisms, including the execution time, method and parameters of watering, fertilizing, pruning and pest control operations, and form a maintenance operation record; Real-time data including temperature, humidity, light intensity, and soil moisture are collected through a sensor network, and supplemented with environmental information by a third-party weather API to form environmental sensor data. Visual features, temporal features, and semantic features are extracted by combining time-series image sequences, maintenance operation records, and environmental sensor data.

3. The training method for the growth milestone recognition model based on multimodal feature fusion according to claim 1, characterized in that, The growth milestone recognition model is based on a multimodal feature fusion architecture. It learns feature maps through a multi-task learning strategy and optimizes the model parameters using a weighted fusion logic of visual, temporal, and semantic features. During the weighting process, weights are dynamically adapted based on organism type, user preferences, and application scenario, including: Based on the requirements of multimodal feature fusion and milestone recognition goals, a fusion architecture of EfficientNet-B4 backbone network + ConvLSTM temporal modeling + YOLOv8 detection head is built to perform hierarchical extraction and feature mapping learning of visual, temporal and semantic features. The feature fusion logic is designed based on a multi-task learning strategy, clarifying the extraction dimensions and fusion ratios of visual features, temporal features, and semantic features, and generating feature fusion rules. Based on the adaptation requirements of life form type, user preferences, and application scenarios, a dynamic weight adjustment mechanism is set up. Plants focus on visual and temporal characteristics, infants and young children focus on semantic and emotional characteristics, scientific report scenarios focus on temporal and semantic characteristics, and emotional short film scenarios focus on semantic and visual characteristics. The system integrates and executes the fusion architecture, feature fusion rules, and dynamic weighting mechanism. Through multi-task learning, it optimizes the parameters of the growth milestone recognition model and generates training result information of the growth milestone recognition model, which includes feature extraction results, weight configuration, and fusion output. This enables efficient fusion of multimodal features and accurate milestone recognition.

4. The training method for the growth milestone recognition model based on multimodal feature fusion according to claim 3, characterized in that, The inference performance of the growth milestone recognition model is optimized by combining organism-specific rules and dynamic density adaptive algorithms. Simultaneously, a temporal NMS post-processing step is employed to suppress redundant frames within a T-time window before and after high-resolution keyframes. This generates inference optimization results information for the growth milestone recognition model, including rule configuration, density parameters, suppression range, and optimization process. Based on the reasoning optimization requirements of the growth milestone recognition model, obtain the parameters of the trained multimodal fusion growth milestone recognition model, organism-specific rules, and dynamic density adaptive algorithm. The initial inference results of the growth milestone identification model are validated to confirm the confidence level of milestone identification and the rationality of key frame selection, and an inference result validation report is generated. Based on the adaptation needs of different life forms, formulate specific rule application standards and generate rule application parameters; Dynamically adjust keyframe density based on content richness, and select density function on a frame-by-frame basis. Weight coefficients are set for content change rate, information entropy, and semantic importance to generate an adaptive density configuration. For content change rate, For information entropy, For semantic importance, , , These are learnable parameters; To address the keyframe temporal clustering problem, a timing NMS processing rule is set, clarifying the suppression range and score adjustment strategy of the T-time window before and after high-scoring keyframes, and avoiding the retention of redundant frames; Integrate rule application parameters, density adaptive configuration, and time-series NMS rules; formulate operational specifications for growth milestone identification model inference optimization; and clarify the rule calling order and parameter adjustment process. Integrate inference result verification reports, rule application parameters, density adaptive configuration, time-series NMS rules and operating specifications to generate growth milestone recognition model inference optimization result information that includes rule configuration, density parameters, suppression range and optimization process.

5. The training method for the growth milestone recognition model based on multimodal feature fusion according to claim 1, characterized in that, The method further includes: Obtain information about the object to be monitored; The growth milestone recognition model is used to process the information of the object to be monitored, and generate milestone recognition results, key frame screening results and supporting auxiliary information for the object to be monitored. The supporting auxiliary information includes disease diagnosis related data, structured growth record data and conversation interaction data.

6. A training device for a growth milestone recognition model based on multimodal feature fusion, characterized in that, The apparatus is configured to perform the training method for the growth milestone recognition model based on multimodal feature fusion as described in any one of claims 1 to 5.

7. An electronic device, characterized in that, include: First processor; and memory for storing executable instructions of the first processor; The first processor is configured to execute the training method of the growth milestone recognition model based on multimodal feature fusion as described in any one of claims 1 to 5 by executing the executable instructions.

8. A computing device comprising a memory for storing computer program instructions and a second processor for executing the computer program instructions, wherein, When the computer program instructions are executed by the second processor, the device is triggered to execute the training method of the growth milestone recognition model based on multimodal feature fusion as described in any one of claims 1 to 5.

Images