A method and system for monitoring behavior and analyzing activity rhythm of a li tree trunk borer based on multi-modal AI

Abstract

The application discloses a litchi borer behavior monitoring and activity rhythm analysis method and system based on a multi-modal AI, and the method comprises the following steps: acquiring continuous video data of litchi borer under night infrared conditions; acquiring time sequence trajectory data keeping the continuity of insect body ID based on the continuous video data and in combination with an improved target detection model; obtaining a behavior segment sequence according to the time sequence trajectory data; acquiring a behavior annotation sequence based on the behavior segment sequence; constructing a behavior space-time database based on the time sequence trajectory data and the behavior annotation sequence; and outputting an activity rhythm analysis result of the litchi borer based on the behavior space-time database and in combination with a night activity rhythm model. The application replaces manual observation with non-contact full-automatic analysis, and realizes continuous and objective rhythm analysis under the premise of not affecting the natural behavior of insects.

Description

Technical Field

[0001] This invention belongs to the fields of agricultural intelligent monitoring and computer vision technology, and in particular relates to a method and system for monitoring the behavior and analyzing the activity rhythm of litchi fruit borers based on multimodal AI. Background Technology

[0002] The litchi fruit borer, belonging to the family Lepidoptera and the family Lepidoptera, is one of the major pests in litchi-growing areas of my country. Currently, its control relies heavily on the frequent and extensive use of chemical pesticides. However, this method is not only ineffective but also causes serious environmental pollution and poses safety risks to agricultural products. Since the litchi fruit borer's important reproductive activities, such as emergence, mating, and oviposition, occur primarily at night during the dark hours, light-repelling technology effectively avoids the drawbacks of chemical control by projecting LED light at specific times at night to interfere with the borer's mating and oviposition behaviors. However, this technology requires continuous nighttime illumination. Field survey data shows that the electricity cost for 40 days of continuous nighttime lighting (11 hours) in orchards during the litchi fruiting period is approximately 55 yuan per mu. Existing research indicates that the mating time for adult litchi fruit borers is only 20-90 minutes. If the peak periods of the borer's nighttime reproductive behavior can be precisely identified, electricity costs can be significantly reduced.

[0003] Therefore, understanding the activity rhythms of the litchi fruit borer is a crucial foundation for developing precise control strategies. Existing research methods can be mainly categorized into the following three types: (1) Manual observation method. Researchers enter the field according to a preset time window and record and statistically analyze the behavior of the target insects by human observation. This method is highly dependent on the experience of researchers, requires a large amount of manpower, and long-term nighttime observation is prone to data bias due to fatigue. In addition, due to the limitations of observation frequency and duration, the amount of data obtained by the manual observation method is limited and the time distribution is discontinuous, resulting in a coarse understanding of the activity patterns of the litchi fruit borer.

[0004] (2) Electronic device-based assisted identification method. This type of method introduces devices such as harmonic radar and acoustic sensors to detect and track insects. However, the radar waves and sound signals emitted by the devices are significantly different from the natural habitat of insects, which may interfere with the natural behavior of insects and affect the representativeness of the statistical results.

[0005] (3) Detection methods based on computer vision. Current research has applied target detection and behavior recognition technologies to insect monitoring, but most existing methods are designed for scenarios with sufficient light, simple backgrounds, large insect size, or fixed traps. No research has been found on monitoring litchi fruit borers based on computer vision. The litchi fruit borer is only 4-5 mm long and its activity is concentrated in low-light environments at night, often obscured by branches and leaves. This is significantly different from the applicable scenarios of existing methods, making it difficult to directly transfer and apply them, resulting in insufficient detection and recognition accuracy. In addition, existing automatic monitoring research mostly stays at the level of single-frame detection or simple trajectory visualization. It lacks a complete method to further organize the detection box, cross-frame insect ID, movement trajectory, speed, displacement, dwell time, and spatial distribution into an activity rhythm model. It is difficult to automatically output decision information such as "when the activity is strongest, in which areas the activity is concentrated, whether the activity is periodic, and which time periods are suitable for prevention and control intervention" from continuous video.

[0006] In summary, existing research methods for analyzing the activity rhythm of litchi fruit borers have limitations such as small sample size, coarse granularity, high labor costs, interference with the natural behavior of insects, and low accuracy in identifying small targets at night, which restrict the effective implementation of precise control strategies. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a method and system for monitoring the behavior and analyzing the activity rhythm of the litchi fruit borer based on multimodal AI. This invention introduces computer vision technology into the field of litchi fruit borer activity rhythm analysis, replacing manual observation with non-contact, fully automated analysis, and achieving continuous and objective rhythm analysis without affecting the insect's natural behavior.

[0008] To achieve the above objectives, this invention provides a method for monitoring the behavior and analyzing the activity rhythm of litchi fruit borers based on multimodal AI, comprising: Acquire continuous video data of litchi fruit borer under nighttime infrared conditions; Based on the continuous video data, combined with the improved target detection model, time-series trajectory data that maintains the continuity of the insect ID is obtained; Based on the time-series trajectory data, a sequence of behavioral segments is obtained; Based on the behavior fragment sequence, obtain the behavior annotation sequence; Based on the aforementioned time-series trajectory data and behavior annotation sequences, a behavior spatiotemporal database is constructed; Based on the aforementioned spatiotemporal database of behavior, combined with the nocturnal activity rhythm model, the activity rhythm analysis results of the litchi fruit borer are output.

[0009] Optionally, the improved target detection model includes a low-light enhancement module, a backbone feature extraction network, a high-resolution multi-scale fusion module, a selective boundary aggregation module, and a Transformer decoding head; The low-light enhancement module is used to adaptively enhance the dark area features of the input image; The backbone feature extraction network is used to extract features at four scales: P2, P3, P4 and P5. P2 is a high-resolution feature layer with a stride of 4, and P3 to P5 are deep feature layers with strides that are doubled in succession. The high-resolution multi-scale fusion module is used to map features of different scales to the same channel dimension and to perform bidirectional cross-scale fusion of features from P2 to P5. The selective boundary aggregation module is used to perform cross-selection with shallow features and deep semantic features as the dominant features respectively, and to process the outputs of the two recalibrated attention units through convolution and channel concatenation to obtain features that fuse boundary details and semantic information. The Transformer decoding head is used to query the output category, bounding box position, and confidence score based on the candidate target.

[0010] Optionally, based on the continuous video data and combined with an improved target detection model, obtaining time-series trajectory data that maintains the continuity of the insect ID includes: Based on the continuous video data, blurry frames and frames without insects are removed, and a clustering algorithm is used to sample the frames to obtain a keyframe dataset. Based on the keyframe dataset, the video frames are size-normalized and data-augmented to construct an object detection dataset. Based on the target detection dataset, the video frames are input into the improved target detection model, and the bounding box and confidence score of the litchi fruit borer are output. Based on the bounding box and confidence score, a multi-stage matching tracking algorithm is used to perform cross-frame association of the insect body to obtain temporal trajectory data that maintains the continuity of the insect body ID.

[0011] Optionally, based on the bounding box and confidence score, a multi-stage matching tracking algorithm is used to perform cross-frame association of the insect body to obtain temporal trajectory data that maintains the continuity of the insect body ID, including: Based on the bounding boxes and confidence scores, the detection boxes are divided into high-confidence groups and low-confidence groups according to their confidence levels; Based on the high-confidence group, a first round of matching is performed with existing trajectories to obtain preliminary matching results; Based on the unmatched trajectories in the preliminary matching results, a matching cost matrix is ​​constructed between the predicted bounding boxes of the unmatched trajectories and the detection boxes in the low-confidence group, and the Hungarian algorithm is used for a second round of matching; for trajectories that are successfully matched in the second round, the trajectory status is updated using the corresponding low-confidence detection boxes; for trajectories that are still unmatched after the second round of matching, they are marked as temporarily lost. Based on the results of the two rounds of matching, update the trajectory status and initialize a new trajectory to obtain time-series trajectory data that maintains the continuity of the insect ID.

[0012] Optionally, obtaining the behavioral segment sequence based on the time-series trajectory data includes: Based on the time-series trajectory data, spatial edges between individuals in the same frame and temporal edges between adjacent frames of the same individual are established to construct a spatiotemporal graph. Based on the time-series trajectory data, smoothing and differencing operations are performed to extract multi-dimensional kinematic node features; Based on the velocity amplitude sequence in the multi-dimensional kinematic node features, the locations of velocity abrupt changes are detected, and a candidate boundary set is generated. Based on the candidate boundary set, a local window is taken with the candidate boundary as the center. The local context representation is aggregated through a self-attention mechanism to predict the boundary offset and obtain the refined boundary position. Based on the refined boundary position, the long trajectory is divided into several variable-length behavior segments, and each variable-length behavior segment is linearly interpolated and resampled to obtain a behavior segment sequence with a fixed number of frames.

[0013] Optionally, obtaining the behavior annotation sequence based on the behavior fragment sequence includes: Based on the behavioral segment sequence, spatial features of each node in the same frame are aggregated to obtain the spatial features of each node. Based on the spatial characteristics of each node, temporal features are extracted in units of behavioral segments to obtain the temporal features of each insect in that behavioral segment. Based on the spatial and temporal features, a behavior representation vector integrating spatiotemporal information is obtained; Based on the behavior representation vector, the behavior segments are classified to obtain a behavior labeling sequence.

[0014] Optionally, based on the behavior representation vector, the behavior segments are classified to obtain a behavior-labeled sequence, including: Spatial stage output is obtained by average pooling along the time dimension based on spatial features, and temporal stage output is obtained based on temporal features. Based on the spatial stage output and the temporal stage output, a weighted residual fusion is performed using learnable scalar coefficients to obtain a behavior representation vector that fuses spatiotemporal information. Based on the behavior representation vector, the behavior category with the highest probability is obtained after classification head and Softmax activation processing, and the behavior label sequence containing insect ID, behavior category, start frame and end frame is output.

[0015] Optionally, based on the time-series trajectory data and behavior annotation sequences, constructing a behavior spatiotemporal database includes: Based on the time-series trajectory data and behavior annotation sequence, time alignment association is performed using insect ID and timestamp as association keys to obtain a status record containing insect ID, timestamp, location coordinates, velocity components and behavior category; Based on the state records, the insect body ID is used as the main index for organizational storage, and a behavioral spatiotemporal database is constructed.

[0016] Optionally, based on the aforementioned spatiotemporal behavioral database and combined with a nocturnal activity rhythm model, the output of the activity rhythm analysis results of the litchi fruit borer includes: Based on the aforementioned spatiotemporal database of behaviors, the occurrence frequency of each behavior category is statistically analyzed using a sliding time window to obtain a time-series curve of behavior frequency. Based on the behavioral frequency time-series curve, a discrete Fourier transform is performed to identify the main rhythm cycle of the behavior, and the shortest continuous time interval is searched by combining the adjustable coverage threshold to output a precise prevention and control intervention window. Based on the behavior-labeled sequences in the behavior spatiotemporal database, a Markov transition matrix between behaviors is constructed to obtain behavior transition probabilities and typical behavior transformation chains. The behavior transition probabilities and typical behavior transformation chains are used to characterize the sequential dependency relationship between different behaviors. The behavioral frequency time-series curve, the main behavioral rhythm cycle, the behavioral transition probability, and the typical behavioral transition chain are used together as the results of the activity rhythm analysis of the litchi fruit borer.

[0017] The present invention also provides a litchi fruit borer behavior monitoring and activity rhythm analysis system based on multimodal AI, including: a litchi fruit borer target detection and trajectory tracking module, a litchi fruit borer key behavior recognition module based on spatiotemporal graph Transformer, and a litchi fruit borer activity rhythm analysis module. The target detection and trajectory tracking module for the litchi fruit borer is used to acquire continuous video data of the litchi fruit borer under nighttime infrared conditions; based on the continuous video data, combined with an improved target detection model, temporal trajectory data that maintains the continuity of the insect's ID is acquired. The litchi stem borer key behavior recognition module based on the spatiotemporal graph Transformer is used to obtain a behavior segment sequence based on the temporal trajectory data; and to obtain a behavior annotation sequence based on the behavior segment sequence. The activity rhythm analysis module of the litchi fruit borer is used to construct a behavior spatiotemporal database based on the time-series trajectory data and behavior annotation sequences; based on the behavior spatiotemporal database and combined with the nighttime activity rhythm model, it outputs the activity rhythm analysis results of the litchi fruit borer.

[0018] Compared with the prior art, the present invention has the following advantages and technical effects: 1. Automation replaces manual labor and eliminates interference: This invention adopts non-contact video acquisition and fully automatic analysis, which eliminates fatigue bias and nighttime operation costs introduced by manual observation, while avoiding interference from electronic equipment signals on the natural behavior of lychee fruit borers, thus ensuring the objectivity and continuity of data.

[0019] 2. A complete analysis chain is formed from raw video to rhythm results: The three modules are sequentially linked, directly outputting the activity rhythm analysis results of the litchi fruit borer and control intervention suggestions from the raw infrared video, forming a complete automated closed-loop system. This invention is the first to systematically introduce computer vision methods into the field of litchi fruit borer activity rhythm analysis, providing continuous and reliable data support and decision-making basis for precise control.

[0020] 3. High-precision detection and tracking of small targets in complex nighttime scenarios: This invention introduces targeted improvement modules to address the challenges of low light, small targets, and foliage obstruction in the nighttime activities of the litchi fruit borer. This significantly improves the detection accuracy and trajectory continuity in complex nighttime environments, and solves the problem that existing computer vision methods are difficult to directly transfer to litchi fruit borer monitoring scenarios.

[0021] 4. Automatic identification of key behaviors of litchi fruit borer: This invention constructs a key behavior identification module for litchi fruit borer based on spatiotemporal graph Transformer, which unifies the modeling of insect movement characteristics and spatial interaction relationships between individuals, and realizes automatic frame-by-frame classification of key behaviors such as mating, filling the gap in existing methods for fine-grained behavior identification of litchi fruit borer, and providing continuous and accurate behavioral semantic sequences for subsequent rhythm analysis.

[0022] 5. Multi-dimensional Activity Rhythm Analysis and Precise Intervention Window Output: This invention quantifies and analyzes the activity patterns of the litchi fruit borer from two dimensions: behavioral rhythm and behavioral shift. Compared with existing sparse manual observation methods, it significantly improves data continuity, temporal resolution, and analytical dimensions. The adjustable coverage threshold mechanism allows for flexible setting of coverage parameters and automatically outputs the shortest precise intervention window, enabling a shift from continuous all-night illumination to precise time-window intervention, effectively reducing the cost of light-based avoidance control. Attached Figure Description

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a structural diagram of a litchi fruit borer behavior monitoring and activity rhythm analysis system based on multimodal AI according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the implementation of module one in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the implementation of module two in this embodiment of the invention. Figure 4 This is a flowchart illustrating the implementation of module three in an embodiment of the present invention. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0026] This embodiment proposes a method for monitoring the behavior and analyzing the activity rhythm of litchi fruit borers based on multimodal AI, which specifically includes the following steps: Acquire continuous video data of litchi fruit borer under nighttime infrared conditions; Based on the continuous video data, combined with the improved target detection model, time-series trajectory data that maintains the continuity of the insect ID is obtained; Based on the time-series trajectory data, a sequence of behavioral segments is obtained; based on the sequence of behavioral segments, a sequence of behavioral annotations is obtained. Based on the aforementioned time-series trajectory data and behavior annotation sequences, a behavior spatiotemporal database is constructed; Based on the aforementioned spatiotemporal database of behavior, combined with the nocturnal activity rhythm model, the activity rhythm analysis results of the litchi fruit borer are output.

[0027] Specifically, addressing the issues of high labor costs in manual observation schemes and interference with insect natural behavior by electronic device-based assisted identification methods, this invention introduces computer vision technology into the field of activity rhythm analysis of the litchi fruit borer. It replaces manual observation with non-contact, fully automated analysis, achieving continuous and objective rhythm analysis without affecting the insect's natural behavior.

[0028] To address the limitations of existing computer vision methods in the nocturnal activity scenarios of the litchi fruit borer and their lack of behavioral category recognition capabilities, this invention introduces targeted improvements at the detection end for three scenarios: low light, small targets, and foliage occlusion. The low-light enhancement module improves the representation of dark area features through adaptive gain adjustment; the high-resolution multi-scale fusion module introduces a high-resolution feature layer to enhance the perception of small targets; and the selective boundary aggregation module improves detection accuracy in occluded scenarios through fine-grained boundary representation. Building upon this foundation, this invention further constructs a key behavior recognition module for the litchi fruit borer based on a spatiotemporal graph Transformer. This module integrates the insect's movement state and spatial interactions between individuals into a graph structure, enabling frame-by-frame automatic classification of key behaviors such as mating, filling the gap in existing methods for fine-grained insect behavior recognition.

[0029] To address the issues of limited sample size and coarse granularity in existing methods for analyzing activity rhythms, this invention constructs a fine-grained activity rhythm modeling system for litchi fruit borers. This system quantifies and analyzes the activity patterns of litchi fruit borers from three dimensions: behavioral rhythm, behavioral shift, and spatial distribution. It automatically outputs precise control and intervention windows, providing a scientific basis for green control measures such as precise irradiation and timed pesticide application. This enables a shift from continuous all-night intervention to precise time-window intervention, effectively reducing control costs.

[0030] Furthermore, the improved target detection model includes a low-light enhancement module, a backbone feature extraction network, a high-resolution multi-scale fusion module, a selective boundary aggregation module, and a Transformer decoding head; The low-light enhancement module is used to adaptively enhance the dark area features of the input image; The backbone feature extraction network is used to extract features at four scales: P2, P3, P4 and P5. P2 is a high-resolution feature layer with a stride of 4, and P3 to P5 are deep feature layers with strides that are doubled in succession. The high-resolution multi-scale fusion module is used to map features of different scales to the same channel dimension and to perform bidirectional cross-scale fusion of features from P2 to P5. The selective boundary aggregation module is used to perform cross-selection with shallow features and deep semantic features as the dominant features respectively, and to process the outputs of the two recalibrated attention units through convolution and channel concatenation to obtain features that fuse boundary details and semantic information. The Transformer decoding head is used to query the output category, bounding box position, and confidence score based on the candidate target.

[0031] Specifically, an improved target detection model, RT-DETR-LHS, is constructed by introducing a low-light enhancement module, a high-resolution multi-scale fusion module, and a selective boundary aggregation module on the baseline model RT-DETR. The model structure of RT-DETR-LHS includes a low-light enhancement module, a backbone feature extraction network, a high-resolution multi-scale fusion module, a selective boundary aggregation module, and a Transformer decoding head. The low-light enhancement module is located at the input end of the backbone feature extraction network and includes a smoothing pre-enhancement unit, a spatial gain branch, a channel gain branch, and a learnable global brightness shift unit, which is used to adaptively enhance the dark area features of the input image. The backbone feature extraction network is used to extract features at four scales: P2, P3, P4, and P5, where P2 is a high-resolution feature layer with a stride of 4. The high-resolution multi-scale fusion module includes a channel mapping unit, a top-down feature fusion path, and a bottom-up feature fusion path, which are used to map features of different scales to the same channel dimension and perform bidirectional cross-scale fusion of features from P2 to P5. The selective boundary aggregation module includes two recalibrated attention units, which use shallow features and deep semantic features as the dominant features for cross-selection, and the outputs of the two recalibrated attention units are processed by convolution and channel concatenation to obtain features that fuse boundary details and semantic information. The Transformer decoding head outputs the category, bounding box position, and confidence score based on the candidate target query.

[0032] Furthermore, based on the continuous video data and combined with the improved target detection model, the temporal trajectory data that maintains the continuity of the insect ID is obtained, including: Based on the continuous video data, blurry frames and frames without insects are removed, and a clustering algorithm is used to sample the frames to obtain a keyframe dataset. Based on the keyframe dataset, the video frames are size-normalized and data-augmented to construct an object detection dataset. Based on the target detection dataset, the video frames are input into the improved target detection model, and the bounding box and confidence score of the litchi fruit borer are output. Based on the bounding box and confidence score, a multi-stage matching tracking algorithm is used to perform cross-frame association of the insect body to obtain temporal trajectory data that maintains the continuity of the insect body ID.

[0033] Furthermore, based on the bounding box and confidence score, a multi-stage matching tracking algorithm is used to perform cross-frame association of the insect body, obtaining temporal trajectory data that maintains the continuity of the insect body ID, including: Based on the bounding boxes and confidence scores, the detection boxes are divided into high-confidence groups and low-confidence groups according to their confidence levels; Based on the high-confidence group, a first round of matching is performed with existing trajectories to obtain preliminary matching results; Based on the unmatched trajectories in the preliminary matching results, a matching cost matrix is ​​constructed between the predicted bounding boxes of the unmatched trajectories and the detection boxes in the low-confidence group, and the Hungarian algorithm is used for a second round of matching; for trajectories that are successfully matched in the second round, the trajectory status is updated using the corresponding low-confidence detection boxes; for trajectories that are still unmatched after the second round of matching, they are marked as temporarily lost. Based on the results of the two rounds of matching, update the trajectory status and initialize a new trajectory to obtain time-series trajectory data that maintains the continuity of the insect ID.

[0034] Furthermore, based on the time-series trajectory data, the resulting sequence of behavioral segments includes: Based on the time-series trajectory data, spatial edges between individuals in the same frame and temporal edges between adjacent frames of the same individual are established to construct a spatiotemporal graph. Based on the time-series trajectory data, smoothing and differencing operations are performed to extract multi-dimensional kinematic node features; Based on the velocity amplitude sequence in the multi-dimensional kinematic node features, the locations of velocity abrupt changes are detected, and a candidate boundary set is generated. Based on the candidate boundary set, a local window is taken with the candidate boundary as the center. The local context representation is aggregated through a self-attention mechanism to predict the boundary offset and obtain the refined boundary position. Based on the refined boundary position, the long trajectory is divided into several variable-length behavior segments, and each variable-length behavior segment is linearly interpolated and resampled to obtain a behavior segment sequence with a fixed number of frames.

[0035] Further, based on the behavior fragment sequence, obtaining the behavior annotation sequence includes: Based on the behavioral segment sequence, spatial features of each node in the same frame are aggregated to obtain the spatial features of each node. Based on the spatial characteristics of each node, temporal features are extracted in units of behavioral segments to obtain the temporal features of each insect in that behavioral segment. Based on the spatial and temporal features, a behavior representation vector integrating spatiotemporal information is obtained; based on the behavior representation vector, the behavior segments are classified to obtain a behavior labeling sequence.

[0036] Furthermore, based on the behavior representation vector, the behavior segments are classified to obtain a behavior labeling sequence including: Spatial stage output is obtained by average pooling along the time dimension based on spatial features, and temporal stage output is obtained based on temporal features. Based on the spatial stage output and the temporal stage output, a weighted residual fusion is performed using learnable scalar coefficients to obtain a behavior representation vector that fuses spatiotemporal information. Based on the behavior representation vector, the behavior category with the highest probability is obtained after classification head and Softmax activation processing, and the behavior label sequence containing insect ID, behavior category, start frame and end frame is output.

[0037] Furthermore, based on the aforementioned time-series trajectory data and behavior annotation sequences, constructing a behavior spatiotemporal database includes: Based on the time-series trajectory data and behavior annotation sequence, time alignment association is performed using insect ID and timestamp as association keys to obtain a status record containing insect ID, timestamp, location coordinates, velocity components and behavior category; Based on the state records, the insect body ID is used as the main index for organizational storage, and a behavioral spatiotemporal database is constructed.

[0038] Furthermore, based on the aforementioned spatiotemporal behavioral database and combined with the nocturnal activity rhythm model, the analysis results of the activity rhythm of the litchi fruit borer are output, including: Based on the aforementioned spatiotemporal database of behaviors, the occurrence frequency of each behavior category is statistically analyzed using a sliding time window to obtain a time-series curve of behavior frequency. Based on the behavioral frequency time-series curve, a discrete Fourier transform is performed to identify the main rhythm cycle of the behavior, and the shortest continuous time interval is searched by combining the adjustable coverage threshold to output a precise prevention and control intervention window. Based on the behavior-labeled sequences in the behavior spatiotemporal database, a Markov transition matrix between behaviors is constructed to obtain behavior transition probabilities and typical behavior transformation chains. The behavior transition probabilities and typical behavior transformation chains are used to characterize the sequential dependency relationship between different behaviors. The behavioral frequency time-series curve, the main behavioral rhythm cycle, the behavioral transition probability, and the typical behavioral transition chain are used together as the results of the activity rhythm analysis of the litchi fruit borer.

[0039] This embodiment also provides a system for monitoring the behavior and analyzing the activity rhythm of litchi fruit borers based on multimodal AI, such as... Figure 1 As shown, it specifically includes: target detection and trajectory tracking module for litchi fruit borer (Module 1), key behavior recognition module for litchi fruit borer based on spatiotemporal graph Transformer (Module 2), and activity rhythm analysis module for litchi fruit borer (Module 3). The target detection and trajectory tracking module for the litchi fruit borer is used to acquire continuous video data of the litchi fruit borer under nighttime infrared conditions; based on the continuous video data, combined with an improved target detection model, temporal trajectory data that maintains the continuity of the insect's ID is acquired. The litchi stem borer key behavior recognition module based on the spatiotemporal graph Transformer is used to obtain a behavior segment sequence based on the temporal trajectory data; and to obtain a behavior annotation sequence based on the behavior segment sequence. The activity rhythm analysis module of the litchi fruit borer is used to construct a behavior spatiotemporal database based on the time-series trajectory data and behavior annotation sequences; based on the behavior spatiotemporal database and combined with the nighttime activity rhythm model, it outputs the activity rhythm analysis results of the litchi fruit borer.

[0040] Specifically, the target detection and trajectory tracking module for the litchi fruit borer aims to identify the litchi fruit borer and its trajectory, providing support for subsequent activity analysis. The specific implementation process is as follows: Figure 2 As shown, it specifically includes: (a) Data Acquisition and Preprocessing: First, the living environment of the litchi fruit borer under nighttime cover of branches and leaves was simulated in a laboratory setting, with unmated insects placed in male-female pairs. Second, an infrared camera was used to capture video information, ensuring sufficient resolution and frame rate. After data acquisition, noisy frames, frames without insects, and other data were removed. K-means clustering was then used to extract keyframes, constructing a dataset for model training and testing.

[0041] (II) Target detection based on RT-DETR-LHS: Based on the baseline model RT-DETR, this module introduces three core improvement modules: Low Light Enhancement (LLE), High Resolution Multi-Scale Fusion (HRMF), and Selective Boundary Aggregation (SBA), which enable high-precision detection in low-light, small target, and foliage occlusion scenarios at night.

[0042] (1) Low Light Enhancement Module (LLE): The LLE module is embedded in the network input to improve effective feature representation under low-light conditions by adaptively enhancing dark area features and suppressing bright area overexposure. The LLE module consists of four components: dark area pre-enhancement transformation, spatial gain modeling, channel gain modeling, and a learnable global brightness shift.

[0043] Dark area pre-enhancement transformation: LLE first performs a lightweight non-linear smoothing transformation on the input features, which enhances the brightness of extremely dark areas while avoiding over-brightness in bright areas, providing more stable input features for subsequent processing. ; in, For input features, For smoothed features, This represents the activation function.

[0044] Spatial gain modeling: LLE then generates a spatial gain map at the same scale as the input through two 3×3 convolutions, emphasizing dark areas and key texture locations, thereby adaptively adjusting local brightness in the spatial dimension: ; in, and This represents two 3×3 convolutions, used for channel compression and channel restoration, respectively. This represents the Sigmoid activation function.

[0045] Channel gain modeling: LLE uses global average pooling and 1×1 convolution to construct lightweight channel gains, allowing each color channel to adaptively contribute to the morphological and textural representation of pests. ; in, This indicates a global average pooling operation.

[0046] Global brightness offset: Spatial gain and channel gain are fused by element-wise multiplication to obtain the final joint gain map: ; Then, a learnable global brightness offset parameter is introduced. : ; in, Constrained within the range of [1,2], this allows for stronger enhancement of dark areas and avoids overexposure of bright areas.

[0047] (2) High-resolution multi-scale fusion module (HRMF): This invention introduces a high-resolution feature layer P2 into the original three-layer detection feature set P3-P5, constructing a four-layer detection feature set P2-P5, which enhances the model's ability to perceive small targets such as borers and effectively alleviates the problem of blurred edges and details caused by multiple downsampling.

[0048] Specifically, P2 directly acquires the shallowest output of the backbone network, aligns it via channel mapping, and then participates in bidirectional fusion: the top-down path propagates high-level semantic information to P2, compensating for the lack of low-level semantics; the bottom-up path passes detailed information from P2 layer by layer to P3-P5, supplementing the spatial information lost by downsampling in deep features. Finally, the four fused features from P2 to P5 are input together into the RT-DETR decoder to achieve end-to-end detection.

[0049] (3) Selective Boundary Aggregation Module (SBA): This invention introduces an SBA module, which improves detection accuracy in scenarios with foliage occlusion and similar target to background by finely characterizing the target contour and recalibrating the target position.

[0050] SBA introduces a novel Recalibrated Attention Unit (RAU). Shallow and deep features are processed by two RAU modules with differentiated mechanisms, forming a complementary relationship: shallow features supplement the detailed information of deep features, and deep features supplement the semantic information of shallow features. Finally, the outputs of the two RAU modules are concatenated along the channel dimension after 3×3 convolution, effectively representing the litchi fruit borer in occluded scenes.

[0051] The input to the RAU module is two sets of features. and First, and The weight maps are obtained after each of them undergoes an independent linear transformation and a Sigmoid activation function. and The conversion process is as follows: ; in, , Two independent The convolution is followed by a projection operation of the Sigmoid activation function.

[0052] Next, the weighted graph and Element-wise multiplication, to achieve Self-filtering. and After element-wise multiplication, then with the inverse weighted graph. Element-wise multiplication, to achieve focus on For components deemed unimportant, implement gating and filtering. Finally, combine the above two items with the original dominant feature. Add them together to get the fused output. .

[0053] ; in, This indicates the negation operation.

[0054] Two RAUs, one using shallow features and the other deep features as the dominant features, perform cross-filtering to achieve complementary fusion of boundary details and semantic information. Correspondingly, the overall SBA process can be represented as: ; in, To integrate the deep semantic features obtained from the third and fourth encoder layers. Derived from the first stage of the main trunk, it contains a wealth of image details. This indicates concatenation along the channel dimension. Final output. This is the fusion result of the SBA module.

[0055] (III) Trajectory generation based on ByteTrack: This invention applies the ByteTrack algorithm to track the trajectory of the litchi fruit borer. This algorithm prioritizes tracking high-confidence bounding boxes while delaying the discarding time of low-confidence bounding boxes, facilitating the retrieval of targets that have temporarily disappeared due to occlusion or other reasons. It effectively reduces trajectory breaks and ID errors, making it suitable for shadowed scenarios in the litchi fruit borer's habitat.

[0056] In actual operation, this module outputs a batch of detection boxes per frame, which are first divided into two groups according to their confidence level. High-confidence boxes are prioritized for the first round of matching with existing trajectories. Trajectories that do not match in the first round are then matched with low-confidence boxes in the second round. This utilizes target information that is still detected despite reduced confidence due to partial occlusion, motion blur, or low contrast, reducing trajectory interruptions and insect ID switching. When no detection box is generated for the target in the current frame, the second round of matching cannot complete trajectory association in that frame, and the corresponding trajectory is temporarily lost. If the target reappears within a preset number of frames and meets the matching conditions, the original trajectory is updated. If the number of consecutive unmatched frames exceeds a preset threshold, the trajectory is terminated. After both rounds of matching are completed, the trajectory status is updated uniformly. Trajectory markers that have not matched for a long time disappear, and newly appearing boxes are initialized as new trajectories. This process is repeated frame by frame to continuously maintain the complete trajectory of each insect.

[0057] The formal definition of the above process is as follows. The target trajectory at time t is... ,in Indicates the location of the target bounding box. Indicates the trajectory confidence level. This represents historical state information. The detector's output at time t is... Each detection box Includes location and confidence level Cascaded Hungarian matching is used for data association, and the matching cost is defined as: ; in for The j-th trajectory at time j, for The first moment One detection box, Calculate the overlap between the two bounding boxes. The lower the cost, the more likely the trajectory and the detection box belong to the same target.

[0058] Successfully matched trajectories are updated in status; unmatched trajectories are moved to a temporary state. Low-confidence detection boxes participate in the second round of matching, making full use of detection information. If a trajectory fails to match for several consecutive frames, it is considered to have disappeared; if a new detection box consistently matches for several consecutive frames, it is initialized as a new trajectory. Trajectory confidence is updated exponentially.

[0059] ; in For smoothing coefficients, The confidence level of the current frame. For historical confidence, This represents the current detection confidence level.

[0060] More specifically, the goal of the key behavior recognition module for the litchi fruit borer based on the spatiotemporal graph Transformer is to perform behavioral semantic analysis on the trajectory data output by Module 1, identify the behavioral category of each litchi fruit borer in any time period, and provide behavioral annotation sequences for subsequent activity rhythm modeling. This module constructs a spatiotemporal graph representation framework, unifying the insect's movement state and the spatial interaction relationships between individuals into the graph structure. After two stages of processing—adaptive behavior segmentation and spatial stage and temporal feature extraction—the data is input into a classification head, which outputs the category prediction for each behavior segment. The specific process is as follows: Figure 3 As shown, the specific steps include: First, modeling the frame-by-frame trajectory data into a spatiotemporal graph and constructing seven-dimensional kinematic node features; then, adaptively segmenting the long trajectory of each insect into behavioral segments, so that each input segment corresponds to a semantically complete single behavior in time; the segmented segments are processed sequentially through spatial and temporal stages, and the outputs of the two stages are weighted and fused before being sent to the classification head to output the behavioral annotation sequence of each insect.

[0061] The specific implementation process is as follows: (I) Spatiotemporal graph construction and multidimensional feature construction: The output of module one is a frame-by-frame position sequence for each insect. , Based on this, this module models the video sequence as a spatiotemporal graph. ,in: Node set Each node Representing the The first frame The state representation of a single insect; all insect nodes within the same frame constitute a spatial snapshot of that frame; spatial edge set Spatial edges are established between any two insects within the same frame to describe the interaction relationships between individuals in the same frame; Time edge set A temporal boundary is established between adjacent frames for the same insect body to describe the motion continuity of an individual across frames.

[0062] The spatial stage involves multi-head self-attention to aggregate cross-individual features along the spatial edge; the temporal stage uses PatchTST to model the feature sequence of each insect along the temporal edge. The two stages are processed sequentially to jointly complete behavior recognition.

[0063] Based on this graph structure, the initial features of each node are composed of the insect's kinematic information, and the quality of these features directly affects the accuracy of subsequent behavior recognition. Therefore, this module introduces a moving average smoothing filter during the feature construction stage. The coordinate sequence of each insect is preprocessed before difference calculation to obtain velocity and acceleration features. This smoothing process effectively suppresses detection noise in the ByteTrack output coordinates, preventing noise from being amplified during the difference calculation and introducing false motion signals, thus ensuring the reliability of subsequent features.

[0064] Let the first Only insects in the first The position of the frame is Based on window size Smoothing with moving average: ; Based on the smoothed coordinates, respectively for the horizontal ( ) and vertical ( The first-order difference yields the velocity component, and the second-order difference yields the acceleration component. Retaining the direction component allows for simultaneous encoding of both velocity and direction of motion. ; ; in , insects In the The horizontal and vertical velocity components of the frame, , These are the acceleration components in the corresponding directions.

[0065] This module additionally introduces the nearest neighbor distance between insects during the feature construction phase. As the seventh dimension of contextual features, this enables the single-insect classifier to perceive the interactive behavior of multiple insects: ; When there is only one insect in the frame Setting it to the preset maximum value indicates no interaction.

[0066] In summary, each node The initial feature vector is seven-dimensional: ; in , These are the smoothed position coordinates. , For velocity components, , For acceleration components, Nearest neighbor distance Represents a seven-dimensional real vector space. (Superscript) This indicates that the feature is the initial input and has not yet undergone update processing by the graph neural network. The velocity and acceleration are then compared... , The directional component is preserved rather than merged into the amplitude, allowing the feature to simultaneously encode both motion rate and direction, thus providing a more complete kinematic description. This seven-dimensional feature is artificially constructed based on insect kinematic analysis, with each dimension having a clear physical meaning, exhibiting good interpretability and convergence efficiency under limited labeled data conditions.

[0067] (ii) Adaptive behavior segmentation: This module proposes a coarse-fine two-stage adaptive segmentation mechanism, which divides long trajectories into variable-length behavior segments based on the abrupt change points of motion states. This ensures that each input segment corresponds to a semantically complete single behavior in time, avoiding the problems of cross-behavior mixing or single behavior truncation caused by fixed-length windows, and providing semantically consistent input for subsequent classification.

[0068] (1) Coarse cutting: With velocity amplitude For the signal, detect locations of abrupt amplitude changes on the time axis and generate a candidate boundary set. When the velocity amplitude change between adjacent frames exceeds an adaptive threshold... When the boundary is determined to be a candidate boundary: ; threshold The value is adaptively determined by the mean and standard deviation of the velocity amplitude change of the insect's complete trajectory: ; in insect Changes in velocity amplitude across all adjacent frames in the complete trajectory The mean, Its standard deviation, This is a hyperparameter. This method adaptively adjusts the threshold according to the overall movement intensity of each insect, avoiding misjudgment between fast and slow individuals when the threshold is fixed.

[0069] The coarse segmentation step has low computational cost and can quickly narrow the search space to near the candidate boundary.

[0070] (2) Fine cutting: This module introduces a lightweight, learnable boundary refinement network for each candidate boundary location. Take it as the center and take the front and back sides Frames constitute a local window, and the feature sequence within the window Through a single-layer self-attention mechanism, the model dynamically captures the changing patterns of motion states in each frame within the window, thus locating the true boundary positions. ; in For window radius hyperparameter, For single-layer multi-head self-attention, This is the local context representation after attention aggregation. Subsequently, a linear regression head is used to predict the boundary offset, resulting in the refined boundary position: ; Offset prediction corrects the systematic lag caused by coordinate smoothing of candidate boundaries, bringing the boundaries back to the true switching moment. Furthermore, joint training of the boundary refinement network and the PatchTST classification head directly aligns the boundary location optimization objective with the behavior classification loss, achieving a synergistic improvement in boundary localization accuracy and classification performance. Final segmentation result. Divide the long trajectory into several variable-length behavior segments.

[0071] (3) Equal-length resampling: Because the duration of each behavior segment obtained from adaptive segmentation varies, their frame counts also differ. This module performs linear interpolation on each segment and resamples them to a fixed number of frames. This addresses the engineering problem of inconsistent input shapes while preserving the relative temporal structure of motion trends within segments, without altering the semantic boundaries established by the segmentation. Specifically, for the target time point... Take the frames that are closest to each other in the original frame sequence. , The interpolation features are obtained by linearly weighting them proportionally. ; in The interpolation ratio reflects the relative position of the target time point between two adjacent original frames. After resampling, each action segment is represented as a shape of... The node feature matrix is ​​input into subsequent processing stages.

[0072] (III) Spatial Feature Extraction: The goal of spatial feature extraction is to integrate the features of each insect into the state information of other insects in the same frame snapshot, so that subsequent temporal modeling can perceive the group interaction background in which the individual is located.

[0073] This module employs multi-head self-attention aggregation to achieve spatial feature aggregation. It eliminates the need for pre-defined neighbor ranges, calculating attention scores for each pair of insect nodes within the same frame. The model autonomously learns which interactions between individuals are more important for current behavior decisions. When there is only one insect in the frame, the model naturally degenerates into a self-transformation of a single node, without introducing additional noise.

[0074] Let the first Intra-frame total Only one insect, the first The initial characteristics of the insect are Through learnable projection matrices Perform a linear mapping on the features, where This represents the feature dimension after projection. (Insect) insect Attention scores are calculated as follows: ; Where the denominator This is a scaling factor. A higher attention score indicates a better performance. With insects The stronger the interaction, the better. After applying Softmax normalization to the attention scores, the features of all insects are weighted and aggregated using the normalized weights to obtain node representations that incorporate spatial interaction information. ; in insect In the The spatial features of a frame carry information about the position and motion state of other individuals in the same frame.

[0075] (iv) Temporal feature extraction: After the spatial phase is completed, each insect will be within the behavioral segment. Frame features constitute a time sequence ,in The frames are derived from the output of adaptive behavior segmentation and equal-length resampling, with each sequence corresponding to a semantically complete behavior segment. The goal of temporal feature extraction is to extract the temporal structural features of the behavior from this sequence, i.e., the evolution of motion patterns over time.

[0076] This module uses PatchTST to model the above sequence. PatchTST divides the temporal sequence into several local segments (patches), each patch corresponding to the motion state of several consecutive frames, and then uses the Transformer self-attention mechanism to capture the long-range temporal dependencies between patches. Compared with frame-by-frame processing, patch partitioning can compress local motion trends into compact semantic units, reduce redundancy, and enhance the perception of the overall structure of behavior.

[0077] The length is The sequence by step size Window size Divided into The nth patch, the Each patch is represented as: ; Apply standard multi-head self-attention to the patch sequence to capture the dependencies between motion patterns at different time intervals: ; right Global average pooling along the time dimension yields the temporal representation vector of this behavioral segment. This serves as a compact description of the insect's movement pattern within this behavioral segment.

[0078] (v) Feature fusion and behavior classification: To comprehensively utilize spatial interaction information and temporal motion information, this module performs weighted residual fusion of the spatial features output by multi-head self-attention aggregation and the temporal features output by PatchTST. Learnable scalar coefficients are introduced. To adaptively adjust the contribution of spatial branches to the final representation: ; in Output for spatial stage Average pooling results along the time dimension Adaptive learning during training. When there is only one insect in the scene... Excluding cross-individual information, The model will tend towards a smaller value during training, and automatically degenerate into a classifier based on temporal features, ensuring consistent applicability of the method in both single-insect and multi-insect scenarios.

[0079] Fusion features After applying a fully connected classification head and Softmax activation, the category with the highest probability is selected as the predicted label for the behavioral segment. The final output of Module 2 is the behavioral label sequence for each insect, in the format of (insect ID, behavioral category, start frame, end frame), which is used by Module 3 for activity rhythm modeling.

[0080] More specifically, the activity rhythm analysis module for the litchi fruit borer uses the location and movement data from Module 1 and the behavioral annotation sequences from Module 2 as input. First, it constructs a spatiotemporal database of behavior, then quantitatively analyzes the activity patterns of the litchi fruit borer from two dimensions: behavioral rhythm and behavioral transfer. Behavioral rhythm analysis directly outputs precise control and intervention windows; behavioral transfer modeling quantitatively characterizes the typical chain of reproductive behavior. These two dimensions together constitute a complete control decision support system, which significantly improves continuity, temporal resolution, and analytical dimensions compared to existing sparse manual observation methods. Figure 4 As shown, it specifically includes: (I) Construction of a behavioral spatiotemporal database: This module associates and fuses the frame-by-frame position and motion data output from Module 1 with the behavior annotation sequence output from Module 2, establishing a spatiotemporal database of behavior with the insect as the primary index. The basic storage unit of the database is a snapshot of the state of a single insect at a single point in time, with fields including insect ID, timestamp, and smoothed coordinates. velocity components The structure binds spatial trajectories and behavioral semantics to the same record, including behavioral categories. This structure supports multi-dimensional slice queries by behavioral category, time window, and spatial region, providing a unified data foundation for subsequent rhythm analysis.

[0081] (II) Behavioral rhythm analysis and prevention window extraction: This module uses a time window sliding statistical method combined with discrete Fourier transform to quantify the frequency of occurrence of various behavioral categories of litchi fruit borers in different time periods, and extracts periodic rhythmic features from them to achieve automatic location of high-incidence periods of key reproductive behaviors such as mating, and output precise prevention and control intervention time windows.

[0082] First, the observation period was divided into equal-length windows. The frequency of each behavior category within each window was then calculated to obtain the time-series curve of behavior frequency. ,in Indicates the category of behavior. For time window indexing, For window size (number of frames). For the first The behavior label corresponding to the frame. This is an indicator function; it takes the value 1 if the condition is true, and 0 otherwise.

[0083] ; This module further incorporates Discrete Fourier Transform (DFT) into the frequency-domain analysis of the behavioral frequency time-series curves to identify the periodic patterns of the behavior. Calculate one by one: ; in This represents the total number of time windows. For frequency component index, The imaginary unit. Each The corresponding period is The components within each time window. Take the amplitude. The maximum point corresponding to The main rhythmic period of this behavior was calculated. The amplitude distribution of each frequency component was visualized using a spectrogram. The more prominent the peak in the spectrogram, the stronger the rhythmic regularity of the behavior in the corresponding period, thus confirming the high-incidence rhythm of key behaviors such as mating of the litchi fruit borer.

[0084] In addition, this module provides an adjustable coverage threshold mechanism for precisely defining the start and end boundaries of the intervention window. Let the specified target coverage be... Search for the shortest continuous time interval on the timeline. This ensures that the cumulative frequency of behavior within that interval is not less than a certain percentage of the total frequency. Times: ; This refers to the recommended intervention window. The smaller the value, the shorter the output window and the lower the intervention cost; The larger the value, the wider the time window and the more comprehensive the coverage. It can be flexibly adjusted according to the needs of epidemic prevention and control. This allows for a dynamic balance between accuracy and coverage, providing a scientific basis for green prevention and control measures such as precise irradiation time and timed drug application, and enabling a shift from continuous all-night intervention to precise time-window intervention, effectively reducing prevention and control costs.

[0085] (III) Behavior transfer modeling: This module constructs Markov transition matrices between behaviors to model the state transition patterns of the litchi fruit borer's behavioral sequences, quantitatively characterizes the typical transition chains between key behaviors such as mating, and reveals the behavioral temporal patterns before and after mating, providing a basis for predicting reproductive behavior.

[0086] Let the set of behavioral states be transition probability Defined as behavior Actions immediately following the occurrence The conditional probability.

[0087] ; in For behavior The behavior that followed immediately The number of times. The transition matrix is ​​presented in tabular form. The larger the value of a cell in the matrix, the more frequent the corresponding behavioral transition. From this, the most typical behavioral chain before and after mating of the lychee fruit borer can be read.

[0088] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for monitoring the behavior and analyzing the activity rhythm of litchi fruit borer based on multimodal AI, characterized in that, include: Acquire continuous video data of litchi fruit borer under nighttime infrared conditions; Based on the continuous video data, combined with the improved target detection model, time-series trajectory data that maintains the continuity of the insect ID is obtained; Based on the time-series trajectory data, a sequence of behavioral segments is obtained; Based on the behavior fragment sequence, obtain the behavior annotation sequence; Based on the aforementioned time-series trajectory data and behavior annotation sequences, a behavior spatiotemporal database is constructed; Based on the aforementioned spatiotemporal database of behavior, combined with the nocturnal activity rhythm model, the activity rhythm analysis results of the litchi fruit borer are output.

2. The method for monitoring the behavior and analyzing the activity rhythm of litchi fruit borers based on multimodal AI according to claim 1, characterized in that, The improved target detection model includes a low-light enhancement module, a backbone feature extraction network, a high-resolution multi-scale fusion module, a selective boundary aggregation module, and a Transformer decoding head; The low-light enhancement module is used to adaptively enhance the dark area features of the input image; The backbone feature extraction network is used to extract features at four scales: P2, P3, P4 and P5. P2 is a high-resolution feature layer with a stride of 4, and P3 to P5 are deep feature layers with strides that are doubled in succession. The high-resolution multi-scale fusion module is used to map features of different scales to the same channel dimension and to perform bidirectional cross-scale fusion of features from P2 to P5. The selective boundary aggregation module is used to perform cross-selection with shallow features and deep semantic features as the dominant features respectively, and to process the outputs of the two recalibrated attention units through convolution and channel concatenation to obtain features that fuse boundary details and semantic information. The Transformer decoding head is used to query the output category, bounding box position, and confidence score based on the candidate target.

3. The method for monitoring the behavior and analyzing the activity rhythm of litchi fruit borers based on multimodal AI according to claim 1, characterized in that, Based on the continuous video data and combined with the improved target detection model, the temporal trajectory data that maintains the continuity of the insect ID is obtained, including: Based on the continuous video data, blurry frames and frames without insects are removed, and a clustering algorithm is used to sample the frames to obtain a keyframe dataset. Based on the keyframe dataset, the video frames are size-normalized and data-augmented to construct an object detection dataset. Based on the target detection dataset, the video frames are input into the improved target detection model, and the bounding box and confidence score of the litchi fruit borer are output. Based on the bounding box and confidence score, a multi-stage matching tracking algorithm is used to perform cross-frame association of the insect body to obtain temporal trajectory data that maintains the continuity of the insect body ID.

4. The method for monitoring the behavior and analyzing the activity rhythm of litchi fruit borers based on multimodal AI according to claim 3, characterized in that, Based on the bounding box and confidence score, a multi-stage matching tracking algorithm is used to perform cross-frame association of the insect body, obtaining temporal trajectory data that maintains the continuity of the insect body ID, including: Based on the bounding boxes and confidence scores, the detection boxes are divided into high-confidence groups and low-confidence groups according to their confidence levels; Based on the high-confidence group, a first round of matching is performed with existing trajectories to obtain preliminary matching results; Based on the unmatched trajectories in the preliminary matching results, a matching cost matrix is ​​constructed between the predicted bounding boxes of the unmatched trajectories and the detection boxes in the low-confidence group, and the Hungarian algorithm is used for the second round of matching; for trajectories that are successfully matched in the second round, the trajectory status is updated using the corresponding low-confidence detection boxes; for trajectories that are still unmatched after the second round of matching, they are marked as temporarily lost. Based on the results of the two rounds of matching, update the trajectory status and initialize a new trajectory to obtain time-series trajectory data that maintains the continuity of the insect ID.

5. The method for monitoring the behavior and analyzing the activity rhythm of litchi fruit borers based on multimodal AI according to claim 1, characterized in that, Based on the time-series trajectory data, the behavioral segment sequence includes: Based on the time-series trajectory data, spatial edges between individuals in the same frame and temporal edges between adjacent frames of the same individual are established to construct a spatiotemporal graph. Based on the time-series trajectory data, smoothing and differencing operations are performed to extract multi-dimensional kinematic node features; Based on the velocity amplitude sequence in the multi-dimensional kinematic node features, the locations of velocity abrupt changes are detected, and a candidate boundary set is generated. Based on the candidate boundary set, a local window is taken with the candidate boundary as the center. The local context representation is aggregated through a self-attention mechanism to predict the boundary offset and obtain the refined boundary position. Based on the refined boundary position, the long trajectory is divided into several variable-length behavior segments, and each variable-length behavior segment is linearly interpolated and resampled to obtain a behavior segment sequence with a fixed number of frames.

6. The method for monitoring the behavior and analyzing the activity rhythm of litchi fruit borers based on multimodal AI according to claim 1, characterized in that, Based on the behavior fragment sequence, obtaining the behavior annotation sequence includes: Based on the behavioral segment sequence, spatial features of each node in the same frame are aggregated to obtain the spatial features of each node. Based on the spatial characteristics of each node, temporal features are extracted in units of behavioral segments to obtain the temporal features of each insect in that behavioral segment. Based on the spatial and temporal features, a behavior representation vector integrating spatiotemporal information is obtained; Based on the behavior representation vector, the behavior segments are classified to obtain a behavior labeling sequence.

7. The method for monitoring the behavior and analyzing the activity rhythm of litchi fruit borers based on multimodal AI according to claim 6, characterized in that, Based on the behavior representation vector, the behavior segments are classified to obtain a behavior labeling sequence including: Spatial stage output is obtained by average pooling along the time dimension based on spatial features, and temporal stage output is obtained based on temporal features. Based on the spatial stage output and the temporal stage output, a weighted residual fusion is performed using learnable scalar coefficients to obtain a behavior representation vector that fuses spatiotemporal information. Based on the behavior representation vector, the behavior category with the highest probability is obtained after classification head and Softmax activation processing, and the behavior label sequence containing insect ID, behavior category, start frame and end frame is output.

8. The method for monitoring the behavior and analyzing the activity rhythm of litchi fruit borers based on multimodal AI according to claim 1, characterized in that, Based on the aforementioned time-series trajectory data and behavior annotation sequences, constructing a behavior spatiotemporal database includes: Based on the time-series trajectory data and behavior annotation sequence, time alignment association is performed using insect ID and timestamp as association keys to obtain a status record containing insect ID, timestamp, location coordinates, velocity components and behavior category; Based on the state records, the insect body ID is used as the main index for organizational storage, and a behavioral spatiotemporal database is constructed.

9. A method for monitoring the behavior and analyzing the activity rhythm of litchi fruit borers based on multimodal AI according to claim 1, characterized in that, Based on the aforementioned spatiotemporal behavioral database and combined with the nocturnal activity rhythm model, the output of the activity rhythm analysis results of the litchi fruit borer includes: Based on the aforementioned spatiotemporal database of behaviors, the occurrence frequency of each behavior category is statistically analyzed using a sliding time window to obtain a time-series curve of behavior frequency. Based on the behavioral frequency time-series curve, a discrete Fourier transform is performed to identify the main rhythm cycle of the behavior, and the shortest continuous time interval is searched by combining the adjustable coverage threshold to output a precise prevention and control intervention window. Based on the behavior-labeled sequences in the behavior spatiotemporal database, a Markov transition matrix between behaviors is constructed to obtain behavior transition probabilities and typical behavior transformation chains. The behavior transition probabilities and typical behavior transformation chains are used to characterize the sequential dependency relationship between different behaviors. The behavioral frequency time-series curve, the main behavioral rhythm cycle, the behavioral transition probability, and the typical behavioral transition chain are used together as the results of the activity rhythm analysis of the litchi fruit borer.

10. A system for monitoring the behavior and analyzing the activity rhythm of litchi fruit borers based on multimodal AI, used to implement the method described in any one of claims 1-9, characterized in that, include: The target detection and trajectory tracking module for litchi fruit borer, the key behavior recognition module for litchi fruit borer based on spatiotemporal graph Transformer, and the activity rhythm analysis module for litchi fruit borer. The target detection and trajectory tracking module for the litchi fruit borer is used to acquire continuous video data of the litchi fruit borer under nighttime infrared conditions; based on the continuous video data, combined with an improved target detection model, temporal trajectory data that maintains the continuity of the insect's ID is acquired. The litchi stem borer key behavior recognition module based on the spatiotemporal graph Transformer is used to obtain a behavior segment sequence based on the temporal trajectory data; and to obtain a behavior annotation sequence based on the behavior segment sequence. The activity rhythm analysis module of the litchi fruit borer is used to construct a behavior spatiotemporal database based on the time-series trajectory data and behavior annotation sequences; based on the behavior spatiotemporal database and combined with the nighttime activity rhythm model, it outputs the activity rhythm analysis results of the litchi fruit borer.

Images