Insect behavior quantification method and device fusing spatial semantic features and spatiotemporal features

Abstract

The application relates to the computer technical field and provides an insect behavior quantification method and device based on space semantic feature and space-time feature fusion. The insect behavior recognition method comprises the following steps: acquiring an insect monitoring video; inputting the insect monitoring video into a multi-behavior recognition model for processing to obtain an insect multi-behavior recognition result; the multi-behavior recognition model is used for extracting local space features from video frames and extracting space-time features from video clips, and performing multi-behavior recognition based on the local space features and the space-time features; the insect multi-behavior recognition result comprises an occurrence time period of each behavior in multiple behaviors of the insect; and the behavior rhythm of the insect is determined based on the insect multi-behavior recognition result. The insect behavior quantification method and device based on space semantic feature and space-time feature fusion can accurately recognize the behavior of small insects with tiny body size, rapid movement and easy overlap, and effectively realize the quantitative analysis of the insect behavior rhythm.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for quantifying insect behavior by fusing spatial semantic features and spatiotemporal features. Background Technology

[0002] Computer vision technology, with its advantage of non-contact recording, has shown great potential in insect behavior recognition. Related computer vision-based insect behavior monitoring methods can effectively identify the behavior of larger, slower-moving insects. However, these methods often fail to accurately distinguish the different behaviors of tiny insects with non-linear movements and frequent occlusion between individuals (such as the tomato leafminer). Summary of the Invention

[0003] This application provides a method and apparatus for quantifying insect behavior by fusing spatial semantic features and spatiotemporal features, in order to solve the shortcomings of related technologies that cannot accurately distinguish the different behaviors of small-sized, fast-moving insects.

[0004] This application provides a method for quantifying insect behavior by fusing spatial semantic features and spatiotemporal features, including: Obtain insect monitoring videos; Insect monitoring videos are input into a multi-behavior recognition model for processing to obtain insect multi-behavior recognition results. The multi-behavior recognition model is used to extract local spatial features from video frames and spatiotemporal features from video segments, and to perform multi-behavior recognition based on local spatial features and spatiotemporal features. The insect multi-behavior recognition results include the time period of each of the various behaviors of the insect. Based on the results of insect multi-behavior recognition, the behavioral rhythms of insects are determined.

[0005] In one embodiment, multiple behaviors include crawling, feeding, flying, and mating.

[0006] In one embodiment, the multi-behavior recognition model includes a two-dimensional convolutional neural network, a three-dimensional convolutional neural network, and an aggregation network. The two-dimensional convolutional neural network is used to extract local spatial features from video frames, the three-dimensional convolutional neural network is used to extract spatiotemporal features from video segments, and the aggregation network is used to determine fused features based on local spatial features and spatiotemporal features.

[0007] In one embodiment, determining fusion features based on local spatial features and spatiotemporal features includes: The spatiotemporal features are compressed along the time dimension and upsampled along the spatial dimension to obtain the processed spatiotemporal features; the spatial dimension of the processed spatiotemporal features is the same as the spatial dimension of the local spatial features. The local spatial features and the processed spatiotemporal features are spliced ​​together along the channel dimension to obtain the spliced ​​features; Channel attention weights are generated based on the splicing features, and the splicing features are adjusted using the channel attention weights to obtain the fused features.

[0008] In one embodiment, the training process of the multi-action recognition model includes: Multiple sample video clips are acquired, each with at least one behavioral label. The behavioral label is used to identify any one of the behaviors: crawling, feeding, flying, and mating.

[0009] Based on multiple sample video segments, the initial recognition model is trained to obtain a multi-behavior recognition model. The initial recognition model is used to extract local spatial features from video frames and spatiotemporal features from video segments, and to perform multi-behavior recognition based on local spatial features and spatiotemporal features.

[0010] In one embodiment, the loss value of the initial recognition model is determined based on the adjusted target probability; the adjusted target probability is determined through the following process: Based on the classification error and bounding box error between the initial recognition results of the initial recognition model and the ground truth annotations, the comprehensive matching score between the initial recognition results and the ground truth annotations is determined; the ground truth annotations include behavior labels and bounding box annotations. The upper bound of the localization quality is determined based on the largest bounding box complete intersection-union ratio in the set of matching results related to the real annotations; The normalized denominator is determined based on the maximum comprehensive matching score in the matching result set. Based on the comprehensive matching score, the upper bound of the positioning quality, and the normalized denominator, the probability of the truly labeled target is adaptively adjusted to obtain the adjusted target probability.

[0011] This application also provides an insect behavior quantification device that fuses spatial semantic features and spatiotemporal features, including: The acquisition module is used to acquire insect monitoring videos; The processing module is used to input insect monitoring videos into a multi-behavior recognition model for processing, and obtain insect multi-behavior recognition results. The multi-behavior recognition model is used to extract local spatial features from video frames and spatiotemporal features from video segments, and to perform multi-behavior recognition based on local spatial features and spatiotemporal features. The insect multi-behavior recognition results include the time period of each of the various behaviors of the insect. The processing module is also used to determine the behavioral rhythm of insects based on the results of multi-behavioral recognition.

[0012] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the insect behavior quantification method that fuses spatial semantic features and spatiotemporal features as described above.

[0013] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the insect behavior quantification method that fuses spatial semantic features and spatiotemporal features as described above.

[0014] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the insect behavior quantification method that fuses spatial semantic features and spatiotemporal features as described above.

[0015] The insect behavior quantification method and apparatus fused with spatial semantic features and spatiotemporal features provided in this application can improve the accuracy of behavior recognition for small insects that are tiny, fast-moving, and prone to overlapping by integrating two-dimensional spatial features and three-dimensional spatiotemporal features into a multi-behavior recognition model. Furthermore, it can accurately determine the behavioral rhythms of insects, enabling quantitative analysis of these rhythms. This not only eliminates subjective bias and fatigue errors from manual observation, achieving all-weather, interference-free monitoring, but also provides extremely reliable and intuitive quantitative data support for optimizing large-scale trapping strategies for agricultural pests and precisely deploying sex pheromone baits in the field. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a method for quantifying insect behavior by fusing spatial semantic features and spatiotemporal features, as provided in an embodiment of this application.

[0018] Figure 2 This is a schematic diagram of the structure of a data acquisition platform provided in an embodiment of this application.

[0019] Figure 3 This is a schematic diagram of the framework of a multi-behavior recognition model provided in an embodiment of this application.

[0020] Figure 4 This is a schematic diagram of the framework of a two-dimensional convolutional neural network provided in an embodiment of this application.

[0021] Figure 5 This is a schematic diagram of the framework of a three-dimensional convolutional neural network provided in an embodiment of this application.

[0022] Figure 6 This is a schematic diagram of the framework of an aggregation network provided in an embodiment of this application.

[0023] Figure 7 This is a schematic diagram of the structure of an insect behavior quantification device that fuses spatial semantic features and spatiotemporal features, provided in an embodiment of this application.

[0024] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] To facilitate understanding of the insect behavior quantification method that fuses spatial semantic features and spatiotemporal features provided in the embodiments of this application, the following is a brief introduction.

[0027] The tomato leafminer is a highly destructive agricultural pest that poses a serious threat to tomato production. High densities of this insect can cause enormous damage to both open-field and modern greenhouse tomato crops, with yield losses potentially reaching 100% under extreme conditions. Understanding the diurnal rhythms of typical behaviors of this insect, such as crawling, foraging, flight, and mating, can help design more effective trapping strategies and provide valuable insights for the application of sex pheromone baits in the field. Traditional methods for monitoring these behaviors of the tomato leafminer rely primarily on manual observation; however, this method is inefficient, subjective, and cannot provide continuous, long-term monitoring data. Therefore, automating the monitoring of the diurnal activity of adult tomato leafminers is crucial. This can not only effectively support integrated pest management research but also help reduce agricultural production's reliance on chemical pesticides.

[0028] In recent years, the rapid advancements in computer vision technology have significantly transformed the methods of monitoring and analyzing insect behavior across various agricultural disciplines. Leveraging its advantage of non-contact recording, computer vision has demonstrated immense application potential in insect behavior recognition. For example, related technologies can replace manual recording by remotely sensing and recording the posture changes and movement trajectories of insects within experimental chambers, combined with computer-aided analysis.

[0029] Currently, deep learning technology is widely used in computer vision-based insect behavior monitoring to automatically learn deep-level features of images, thereby improving the robustness and accuracy of the model. For example, related technologies can realize dynamic insect behavior recognition based on deep learning and image technology. By extracting and detecting the Region of Interest (ROI) in the insect's head and tail areas, it can effectively avoid misjudgments caused by vibration interference and reduce manual observation time.

[0030] Although the monitoring methods based on computer vision and deep learning perform well in recognizing the behavior of larger, slower-moving insects, their core mechanism mainly relies on extracting two-dimensional spatial features of the insect's behavioral area for recognition, making it difficult to effectively monitor the tomato leafminer.

[0031] Specifically, the spatial feature extraction methods of these approaches are easily limited by physical conditions. Due to the small size of adult tomato leafminer moths, their non-linear and rapid movements, and the frequent occlusion between individuals, existing detection models are prone to blurred, lost, or false detections of behavioral targets, significantly increasing the difficulty of behavioral recognition.

[0032] Furthermore, these methods lack effective representation of spatiotemporal continuity. The typical behavior of adult tomato leafminer moths exhibits clear temporal continuity and dynamic changes, and the determination of their behavioral patterns often requires combining the changes between consecutive frames. Related technologies rely solely on the two-dimensional spatial features of a single frame, making it difficult to fully represent the spatiotemporal process of their action evolution, thus easily leading to inaccurate behavior differentiation or severely insufficient recognition stability.

[0033] Based on this, embodiments of this application provide a method for quantifying insect behavior by fusing spatial semantic features and spatiotemporal features. This method improves the accuracy of behavior recognition for small insects that are tiny, fast-moving, and prone to overlapping by integrating two-dimensional spatial features and three-dimensional spatiotemporal features into a multi-behavior recognition model. Furthermore, it can accurately determine the behavioral rhythms of insects, enabling quantitative analysis of these rhythms. This not only eliminates subjective bias and fatigue errors from manual observation, achieving all-weather, interference-free monitoring, but also provides extremely reliable and intuitive quantitative data support for optimizing large-scale trapping strategies for agricultural pests and precisely deploying sex pheromone baits in the field.

[0034] The following is combined Figures 1-6 This application describes a method for quantifying insect behavior by fusing spatial semantic features and spatiotemporal features, as provided in its embodiments.

[0035] Figure 1 This is a flowchart illustrating the insect behavior quantification method based on the fusion of spatial semantic features and spatiotemporal features provided in this application embodiment. Figure 1 As shown, the method includes the following: Step 110: Obtain insect monitoring videos.

[0036] Among them, insect monitoring videos refer to video data obtained through continuous monitoring of insects. The insects can be adult tomato leafminers. For example, a data acquisition platform can be built using equipment such as infrared cameras, insect rearing cages, and infrared supplemental lighting, and adult tomato leafminers in the insect rearing cage environment can be continuously monitored to obtain continuous video data of the adult tomato leafminers' activities, such as 24-hour uninterrupted video data.

[0037] In one embodiment, in order to ensure the efficiency and standardization of model processing, the collected raw video data can be preprocessed and edited to divide the long video into multiple continuous video segments suitable for model processing, and all video segments have a uniform fixed frame rate and fixed duration.

[0038] As an example, Figure 2 This is a schematic diagram of a data acquisition platform provided in an embodiment of this application. To obtain high-quality insect activity data that conforms to actual application scenarios, a standardized data acquisition platform can be built. Specifically, an insect rearing cage with a size matching the insect's body size can be built, such as the one shown in the figure, which is 15 centimeters (cm) in length, width, and height. The top of the cage can be covered with a highly transparent film to reduce external environmental interference. Adult tomato leafminer moths, the monitoring target, can be placed inside the cage. An infrared camera can be installed directly above the cage to capture video of the tomato leafminer moth's behavior at high resolution and a stable frame rate. Simultaneously, to simulate a real living environment and achieve uninterrupted day and night monitoring, defatted cotton balls soaked in 10% honey water can be placed inside the cage as a nutrient source, and multiple infrared supplementary lights (such as...) can be arranged around the top of the cage. Figure 2 Four of them are distributed in the middle, so as to provide lighting conditions that do not disturb the natural behavior of insects, thus successfully collecting continuous videos containing diurnal rhythm behavior.

[0039] Furthermore, the infrared camera can be connected to an external computer via a data cable to transmit the acquired video data to the computer for storage in real time, where it will then perform subsequent deep learning-based multi-behavior recognition processing. The entire data acquisition platform (including the computer, infrared camera, and other equipment) can be powered by a unified power supply to ensure the long-term continuous operation of the monitoring system. Moreover, the insect density in the rearing cages can be flexibly set according to experimental or practical needs (e.g., 5 or 10 pairs of male and female tomato leafminers), ensuring that the acquired insect monitoring video data is sufficiently representative and supports generalized scenarios.

[0040] Step 120: Input the insect monitoring video into the multi-behavior recognition model for processing to obtain the insect multi-behavior recognition results.

[0041] Among them, the multi-behavior recognition model is used to extract local spatial features from video frames and spatiotemporal features from video segments, and to perform multi-behavior recognition based on local spatial features and spatiotemporal features.

[0042] The results of insect multi-behavior identification include the timing of each of the various behaviors of the insect. These behaviors include crawling, feeding, flying, and mating.

[0043] It should be noted that, considering the small size, non-linear rapid movement, and frequent occlusion between individuals of insects (such as the tomato leafminer), relying solely on the spatial features of a single frame is insufficient to fully characterize their complex movement evolution process and accurately identify insect behavior. Therefore, the multi-behavior recognition model designed in this application combines local spatial features in video frames and spatiotemporal features in video segments for multi-behavior recognition. In this way, the multi-behavior recognition model can not only identify the precise location bounding box of the insect in the video frame, but also accurately identify whether the insect is crawling, feeding, flying, or mating, as well as the corresponding time periods (i.e., start and end times) of these behaviors in the video.

[0044] Step 130: Based on the results of insect multi-behavior recognition, determine the insect's behavioral rhythm.

[0045] In one embodiment, the insect's activity level, such as the average hourly activity level, can be calculated for each behavior based on the time period of occurrence of each behavior in the insect multi-behavior identification results. Furthermore, the circadian rhythm analysis results of the target insect can be determined based on the insect's activity level over multiple consecutive preset time periods.

[0046] In one embodiment, based on the results of insect multi-behavior identification, continuous 24-hour monitoring and quantitative behavioral analysis of insects can be conducted. Specifically, the frequency and duration of four typical behaviors—crawling, feeding, flying, and mating—of each insect (or each group of insects) at different time periods can be statistically analyzed to calculate their average hourly activity level for each behavior. In this way, the diurnal rhythm of insects can be clearly outlined through such long-term data statistics. For example, quantitative analysis can show that insects exhibit significant mating and feeding activities in the morning (e.g., 09:00 to 13:00), or significant crawling, feeding, and flying activities from night to the early morning of the next day (e.g., 16:00 to 09:00 the next day), exhibiting typical nocturnal characteristics with a generally high level of activity.

[0047] Based on this, a multi-behavior recognition model that integrates two-dimensional spatial features and three-dimensional spatiotemporal features can improve the accuracy of behavior recognition for small insects that are tiny, fast-moving, and prone to overlapping. Furthermore, it can accurately determine the behavioral rhythms of insects, enabling quantitative analysis of these rhythms. This not only eliminates subjective bias and fatigue errors from manual observation, allowing for all-weather, interference-free monitoring, but also provides extremely reliable and intuitive quantitative data support for optimizing large-scale trapping strategies for agricultural pests and precisely deploying sex pheromone baits in the field.

[0048] In one embodiment, the multi-behavior recognition model may include two branches: a two-dimensional convolutional neural network and a three-dimensional convolutional neural network.

[0049] Two-dimensional convolutional neural networks are used to extract local spatial features from video frames. They can extract fine-grained local spatial semantic information from key frames of video segments, enabling precise spatial localization and body feature capture of insects.

[0050] Three-dimensional convolutional neural networks are used to extract spatiotemporal features from video clips. They can model time-series video clips containing context to characterize the changes in the movement patterns of insects in the time dimension.

[0051] As an example, a multi-action recognition model can be built based on the recognition model Mamba-YOWO. A two-dimensional convolutional neural network (2D-CNN) can be built using Mamba Out. A three-dimensional convolutional neural network (3D-CNN) can be built using 3D-ShuffleNetV2. Alternatively, the multi-action recognition model, 2D-CNN, and 3D-CNN can be built in other ways without restriction.

[0052] Furthermore, the multi-behavior recognition model may also include an aggregation network. The aggregation network is used to determine fused features based on local spatial features and spatiotemporal features. Specifically, the aggregation network can be constructed based on channel fusion and channel attention mechanisms to deeply fuse local spatial features in video frames and spatiotemporal features in video segments, thereby enhancing the complementarity and discriminability of spatial and spatiotemporal information fusion.

[0053] In one embodiment, when the aggregation network determines the fusion feature based on local spatial features and spatiotemporal features, it can compress the spatiotemporal features along the time dimension and upsample them along the spatial dimension to obtain processed spatiotemporal features. Then, it concatenates the local spatial features and the processed spatiotemporal features along the channel dimension to obtain concatenated features. Channel attention weights are then generated based on the concatenated features, and these weights are used to adjust the concatenated features to obtain the fusion feature. The spatial dimension of the processed spatiotemporal features is the same as the spatial dimension of the local spatial features.

[0054] As an example, Figure 3 This is a schematic diagram of the framework of a multi-behavior recognition model provided in an embodiment of this application. For example... Figure 3 As shown, the model's input is divided into two branches. One branch receives a sequence of input frames (3×D×H×W) containing continuous context (where D represents the number of frames, and H and W represent spatial height and width), which is fed into a 3D CNN for processing to extract spatiotemporal feature information, outputting a 3D-CNN feature map with tensor dimensions represented as C'×H'×W'. The other branch receives selected input keyframes (3×H×W), which are fed into a 2D CNN for processing to extract spatial feature information, outputting a 2D-CNN feature map with tensor dimensions represented as C''×H'×W'. Subsequently, these two feature sets are fed into a channel fusion and attention module (i.e., an aggregation network) for deep interaction and fusion. The fused features are finally decoded by a prediction layer, outputting classification results for insect behavior categories and bounding box regression results for insect target localization.

[0055] As an example, Figure 4 This is a schematic diagram of a two-dimensional convolutional neural network framework provided in an embodiment of this application. Figure 4 As shown, this network can employ the Mamba Out network framework. Specifically, after inputting an image of size H×W×3, it first undergoes initial feature extraction through a front-end layer (i.e., the Stem layer). Subsequently, the network is divided into four stages with decreasing scale: Stage 1, Stage 2, Stage 3, and Stage 4. Each stage includes a gated convolutional neural network (Gated CNN) module for feature extraction and a downsampling operation. As the stages deepen, the size of the output feature maps gradually decreases, while the number of channels gradually increases, with the dimensionality evolving sequentially. , , as well as . Figure 4The right side further illustrates the internal structure of the Gated CNN module. This module employs a residual branching design and includes multiple linear layers, convolutional layers, and normalization layers to enhance feature representation, extract spatial features, stabilize the network, and reinforce important information. Features are gated through element-wise multiplication between branches, and specific states or activation markers are added during merging. Finally, residual connections are established through addition operations, resulting in the output that efficiently extracts fine-grained spatial semantic features.

[0056] As an example, Figure 5 This is a schematic diagram of the framework of a three-dimensional convolutional neural network provided in an embodiment of this application. Figure 5 As shown, the network mainly consists of basic units and downsampling units. In the basic unit, the input features are first split into two branches by channel splitting. The lower branch remains unchanged (as a residual), while the upper branch sequentially undergoes 3D convolution (Conv3d), batch normalization with modified linear units (BN+ReLU), 3D depthwise convolution (DWConv3d), batch normalization (BN), and then Conv3d and BN+ReLU again. The features from the two branches are then concatenated and output after channel rearrangement to facilitate information exchange between channels. In the downsampling unit, to reduce spatial dimensionality, both branches participate in downsampling. The upper branch sequentially passes through Conv3d, BN+ReLU, a depthwise convolution with stride (DWConv3d, stride=2), BN, and then Conv3d and BN+ReLU. The lower branch sequentially passes through DWConv3d, BN, and then Conv3d and BN+ReLU. This reduces information loss during downsampling, maintains dependencies between channels, and enhances the spatiotemporal representation of motion pattern changes. Finally, the features from the two branches are concatenated and rearranged through channels, thereby capturing the dynamic spatiotemporal patterns of short video clips while reducing computational cost.

[0057] As an example, Figure 6 This is a schematic diagram of the framework of an aggregation network provided in an embodiment of this application. For example... Figure 6 As shown, this aggregation network mainly consists of two parts: channel fusion and channel attention modules. In the channel fusion stage, the input consists of two tensors, namely the spatiotemporal features from the 3D-CNN branch (the squares in the diagram are labeled 3D). And spatial features from the 2D-CNN branch (the squares in the diagram are labeled 2D). .first, Compression along the time dimension yields .Then, Upsampled to spatial dimension and Matching, generating After aligning in the time dimension and upsampling in the spatial dimension, and The features are concatenated along the channel dimension to form a concatenated tensor composed of both 3D and 2D features. This concatenated tensor is represented as follows: .

[0058] Subsequently, the concatenated tensor enters the channel attention module, where it is processed by both max pooling and average pooling to extract two different spatial context descriptors. These two descriptors are then fed into a shared multilayer perceptron for dimensionality reduction and expansion. The outputs are aggregated through element-wise summation and then passed through an activation function to generate the final channel attention weight matrix. This weight matrix is ​​applied to the original concatenated tensor, readjusting the feature importance by channel, ultimately outputting a fused and complementary feature map. This enhances the model's ability to identify complex and similar behaviors.

[0059] In one embodiment, the training process of the multi-behavior recognition model includes: acquiring multiple sample video segments; training an initial recognition model based on the multiple sample video segments to obtain the multi-behavior recognition model. Each sample video segment possesses at least one behavior label. The behavior label is used to identify any one of the behaviors: crawling, feeding, flying, and mating. The initial recognition model is used to extract local spatial features from video frames and spatiotemporal features from video segments, and to perform multi-behavior recognition based on the local spatial features and spatiotemporal features.

[0060] Specifically, it can be achieved through the above Figure 2 The data acquisition platform shown continuously collects video data on the activities of insects in an insect rearing cage environment. The collected video data is then edited to retain video clips containing four typical behaviors: crawling, feeding, flying, and mating. All clips have a uniform fixed frame rate (e.g., 30 frames per second (FPS)) and duration (e.g., 8 seconds). The acquired video clips can then be annotated frame by frame to construct multiple sample video clips with behavior labels conforming to the UCF format, and divided into training and test sets in an 8:2 ratio. Furthermore, after setting the initial parameters of the initial recognition model, the training set can be input into the initial recognition model for training. After training, the training weights are obtained, thus configuring a multi-behavior recognition model.

[0061] In one embodiment, three groups of samples can be set up, each containing newly emerged male and female insects at different densities, with the number of male and female pairs being 2 (2:2), 5 (5:5), and 10 (10:10), respectively. Video data is continuously collected for each group of samples for 7 days to obtain the raw video data.

[0062] In one embodiment, multiple video segments can be selected and labeled from video data of tomato leafminer adults with an insect density of 5 pairs, serving as training and evaluation data for the initial recognition model. Video data of tomato leafminer adults with insect densities of 2 and 10 pairs can be used to evaluate the model's generalization ability under low and high density conditions.

[0063] In one embodiment, the loss value of the initial recognition model is determined based on the adjusted target probability. The adjusted target probability is determined through the following process: based on the classification error and bounding box error between the initial recognition result and the ground truth annotation, a comprehensive matching score between the initial recognition result and the ground truth annotation is determined; based on the maximum complete intersection-union ratio of bounding boxes in the matching result set related to the ground truth annotation, an upper bound for localization quality is determined; based on the maximum comprehensive matching score in the matching result set, a normalized denominator is determined; and based on the comprehensive matching score, the upper bound for localization quality, and the normalized denominator, the target probability of the ground truth annotation is adaptively adjusted to obtain the adjusted target probability.

[0064] The initial identification results may include behavior categories and bounding box identification results. Ground truth annotation includes behavior labels and bounding box annotations. Bounding box annotations are the ground truth bounding boxes that annotate individual insects in the video frames.

[0065] Specifically, the loss function used to train the initial recognition model can incorporate Task Alignment Learning (TAL) loss for label assignment and optimized training. Specifically, the initial recognition result set is denoted as... The set of real labeled data is denoted as Define the initial recognition result. With real labeling The similarity measurement function between them is: .

[0066] in, This represents the overall matching score between the initial identification result and the actual annotation. The classification error is represented by the binary cross-entropy loss function. calculate. To identify the probability of the category. For true category labels. To represent the bounding box error, use calculate. To identify bounding boxes. This represents the true bounding box. and is a weighting coefficient used to adjust the contribution of classification error and bounding box error to the similarity metric.

[0067] For each From all Select The top k highest initial recognition results are matched, but each It can only be with one at most Match. If a certain Appearing in multiple From the first k lists, then... With envoy The largest value Perform a match. Furthermore, only matches meeting the following conditions will be considered. Its receptive field center is located in the corresponding Within the bounding box, and from the center of the receptive field... The distance between the center of the bounding box does not exceed the preset radius threshold.

[0068] Furthermore, considering that under normal circumstances, when and After the matching is completed, Considered The target is defined, and a target probability is set for the corresponding category; that is, if it appears, the target probability is 1, and if it does not appear, the target probability is 0. In this embodiment, to introduce the concept of quality-weighted loss (i.e., Qualified Loss), the target probability of the appearing category is adaptively adjusted, defined as: ; ; .

[0069] in, Indicates the first The target probability of class, i.e., the first The dynamic target probability corresponding to each real label. M Indicates and Related matching sets. Indicates the first A real label. Indicates and The prediction results of the completed match. Indicates in Zhongyu With the largest The matching results of the values ​​are used to characterize the upper bound of the positioning quality. Indicates in The largest The matching results of the values ​​are used to normalize the overall matching score. Through the above definition, it is possible to... The value range falls within [0,1], and when When it is larger, The larger the value, the higher the supervision weight of high-quality matching.

[0070] In one embodiment, when training the initial recognition model, the number of training rounds can be set to 10, the batch size to 8, the initial learning rate to 0.001, and the clip lengths to 12 frames. This allows for stable convergence of the model during training, avoids overfitting, fully captures the temporal continuity of insect movements, and achieves a good balance between computational resource overhead and training efficiency.

[0071] In one embodiment, the training process of the multi-behavior recognition model further includes: acquiring multiple sets of video clips; inputting the multiple sets of video clips into the multi-behavior recognition model for processing to obtain recognition results corresponding to each set of video clips; and determining the generalization ability evaluation result of the multi-behavior recognition model based on the recognition results and behavior labels corresponding to each set of video clips. Different sets of video clips correspond to different insect densities. The generalization ability evaluation result is used to indicate the recognition accuracy of the multi-behavior recognition model under different insect densities.

[0072] Specifically, when acquiring multiple sets of video clips, experimental groups containing different numbers of male and female insects can be set up. For example, videos of adult tomato leafminer moths with 2 pairs (2 males and 2 females, representing low density), 5 pairs (representing medium to high density), and 10 pairs (representing high density) can be collected. In the actual training and evaluation process, video clips can be selected from the medium-density (5 pairs) video data for annotation, serving as the main training and basic evaluation data for the multi-behavior recognition model. Simultaneously, low-density (2 pairs) and high-density (10 pairs) video data that have not been trained on the model can be used as independent validation samples, input into the trained multi-behavior recognition model for cross-scene generalization ability testing.

[0073] Based on the model's recognition results and corresponding behavior labels for multiple video segments, the overall mean average precision (MAP) of the model under different density conditions is calculated. For example, the evaluation results show that the overall MAP of the model under 2-pair, 5-pair, and 10-pair conditions are 92.6%, 89.2%, and 65.7%, respectively.

[0074] Based on this, by introducing independent tests under different population density conditions, we can further verify that the multi-behavior recognition model not only maintains high accuracy in normal environments, but also maintains excellent robustness, stability and environmental adaptability in complex cage environments, when faced with extreme interference such as rapid changes in population density and frequent severe occlusion between individuals at high densities.

[0075] In the embodiments described above, a multi-behavior recognition model with 2D-CNN and 3D-CNN branches can capture fine-grained spatial semantic information from key image frames and extract spatiotemporal features from video clips, achieving accurate multi-behavior recognition of insects. Compared to traditional insect behavior recognition methods, the insect behavior quantification method provided in this embodiment can achieve higher detection accuracy while maintaining efficient recognition. It not only supports consistent and objective behavior monitoring but also reduces subjective bias caused by manual observation and minimizes interference from contact monitoring on normal insect behavior. This provides reliable data support for the monitoring and analysis of insect behavioral rhythms, effectively supporting the optimization of large-scale trapping strategies and the field deployment of sex pheromone trapping.

[0076] The following describes the insect behavior quantification device that fuses spatial semantic features and spatiotemporal features according to the embodiments of this application. The insect behavior quantification device that fuses spatial semantic features and spatiotemporal features described below can be referred to in correspondence with the insect behavior quantification method that fuses spatial semantic features and spatiotemporal features described above.

[0077] Figure 7 This is a schematic diagram of the structure of an insect behavior quantification device that fuses spatial semantic features and spatiotemporal features, as provided in an embodiment of this application. Figure 7 As shown in the figure, this application provides a device for quantifying insect behavior by fusing spatial semantic features and spatiotemporal features, including: Module 710 is used to acquire insect monitoring videos; The processing module 720 is used to input insect monitoring videos into a multi-behavior recognition model for processing to obtain insect multi-behavior recognition results. The multi-behavior recognition model is used to extract local spatial features from video frames and spatiotemporal features from video segments, and to perform multi-behavior recognition based on local spatial features and spatiotemporal features. The insect multi-behavior recognition results include the time period of occurrence of each of the various behaviors of the insect. The processing module 720 is also used to determine the behavioral rhythm of insects based on the results of insect multi-behavior recognition.

[0078] In one embodiment, multiple behaviors include crawling, feeding, flying, and mating.

[0079] In one embodiment, the multi-behavior recognition model includes a two-dimensional convolutional neural network, a three-dimensional convolutional neural network, and an aggregation network. The two-dimensional convolutional neural network is used to extract local spatial features from video frames, the three-dimensional convolutional neural network is used to extract spatiotemporal features from video segments, and the aggregation network is used to determine fused features based on local spatial features and spatiotemporal features.

[0080] In one embodiment, determining fusion features based on local spatial features and spatiotemporal features includes: The spatiotemporal features are compressed along the time dimension and upsampled along the spatial dimension to obtain the processed spatiotemporal features; the spatial dimension of the processed spatiotemporal features is the same as the spatial dimension of the local spatial features. The local spatial features and the processed spatiotemporal features are spliced ​​together along the channel dimension to obtain the spliced ​​features; Channel attention weights are generated based on the splicing features, and the splicing features are adjusted using the channel attention weights to obtain the fused features.

[0081] In one embodiment, the training process of the multi-action recognition model includes: Multiple sample video clips are acquired, each with at least one behavioral label. The behavioral label is used to identify any one of the behaviors: crawling, feeding, flying, and mating.

[0082] Based on multiple sample video segments, the initial recognition model is trained to obtain a multi-behavior recognition model. The initial recognition model is used to extract local spatial features from video frames and spatiotemporal features from video segments, and to perform multi-behavior recognition based on local spatial features and spatiotemporal features.

[0083] In one embodiment, the loss value of the initial recognition model is determined based on the adjusted target probability; the adjusted target probability is determined through the following process: Based on the classification error and bounding box error between the initial recognition results of the initial recognition model and the ground truth annotations, the comprehensive matching score between the initial recognition results and the ground truth annotations is determined; the ground truth annotations include behavior labels and bounding box annotations. The upper bound of the localization quality is determined based on the largest bounding box complete intersection-union ratio in the set of matching results related to the real annotations; The normalized denominator is determined based on the maximum comprehensive matching score in the matching result set. Based on the comprehensive matching score, the upper bound of the positioning quality, and the normalized denominator, the probability of the truly labeled target is adaptively adjusted to obtain the adjusted target probability.

[0084] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a method for quantifying insect behavior by fusing spatial semantic features and spatiotemporal features. This method includes: Acquire insect monitoring videos; input the insect monitoring videos into a multi-behavior recognition model for processing to obtain insect multi-behavior recognition results; the multi-behavior recognition model is used to extract local spatial features from video frames and spatiotemporal features from video segments, and to perform multi-behavior recognition based on local spatial features and spatiotemporal features; the insect multi-behavior recognition results include the time period of each of the various behaviors of the insect; based on the insect multi-behavior recognition results, determine the insect's behavioral rhythm.

[0085] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the insect behavior quantification method that fuses spatial semantic features and spatiotemporal features provided by the above methods. The method includes: Acquire insect monitoring videos; input the insect monitoring videos into a multi-behavior recognition model for processing to obtain insect multi-behavior recognition results; the multi-behavior recognition model is used to extract local spatial features from video frames and spatiotemporal features from video segments, and to perform multi-behavior recognition based on local spatial features and spatiotemporal features; the insect multi-behavior recognition results include the time period of each of the various behaviors of the insect; based on the insect multi-behavior recognition results, determine the insect's behavioral rhythm.

[0087] Furthermore, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for quantifying insect behavior by fusing spatial semantic features and spatiotemporal features as provided by the methods described above. This method includes: Acquire insect monitoring videos; input the insect monitoring videos into a multi-behavior recognition model for processing to obtain insect multi-behavior recognition results; the multi-behavior recognition model is used to extract local spatial features from video frames and spatiotemporal features from video segments, and to perform multi-behavior recognition based on local spatial features and spatiotemporal features; the insect multi-behavior recognition results include the time period of each of the various behaviors of the insect; based on the insect multi-behavior recognition results, determine the insect's behavioral rhythm.

[0088] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for quantifying insect behavior by fusing spatial semantic features and spatiotemporal features, characterized in that, include: Obtain insect monitoring videos; The insect monitoring video is input into a multi-behavior recognition model for processing to obtain insect multi-behavior recognition results. The multi-behavior recognition model is used to extract local spatial features from video frames and spatiotemporal features from video segments, and to perform multi-behavior recognition based on the local spatial features and the spatiotemporal features. The insect multi-behavior recognition results include the time period of each of the various behaviors of the insect. Based on the results of the insect multi-behavior recognition, the insect's behavioral rhythm is determined.

2. The method for quantifying insect behavior according to claim 1, characterized in that, The various behaviors include crawling, feeding, flying, and mating.

3. The method for quantifying insect behavior according to claim 1 or 2, characterized in that, The multi-behavior recognition model includes a two-dimensional convolutional neural network, a three-dimensional convolutional neural network, and an aggregation network. The two-dimensional convolutional neural network is used to extract local spatial features from video frames, the three-dimensional convolutional neural network is used to extract spatiotemporal features from video segments, and the aggregation network is used to determine fused features based on the local spatial features and the spatiotemporal features.

4. The method for quantifying insect behavior according to claim 3, characterized in that, The determination of fusion features based on the local spatial features and the spatiotemporal features includes: The spatiotemporal features are compressed along the time dimension and upsampled along the spatial dimension to obtain the processed spatiotemporal features; the spatial dimension of the processed spatiotemporal features is the same as the spatial dimension of the local spatial features. The local spatial features and the processed spatiotemporal features are spliced ​​together along the channel dimension to obtain the spliced ​​features; Channel attention weights are generated based on the splicing features, and the splicing features are adjusted using the channel attention weights to obtain the fused features.

5. The method for quantifying insect behavior according to claim 1 or 2, characterized in that, The training process of the multi-action recognition model includes: Multiple sample video clips are acquired, each of which has at least one behavior tag, which is used to identify any one of the behaviors of crawling, feeding, flying and mating. Based on the multiple sample video segments, the initial recognition model is trained to obtain the multi-behavior recognition model. The initial recognition model is used to extract local spatial features from video frames and spatiotemporal features from video segments, and to perform multi-behavior recognition based on the local spatial features and spatiotemporal features.

6. The method for quantifying insect behavior according to claim 5, characterized in that, The loss value of the initial recognition model is determined based on the adjusted target probability; the adjusted target probability is determined through the following process: Based on the classification error and bounding box error between the initial recognition result of the initial recognition model and the ground truth annotation, a comprehensive matching score between the initial recognition result and the ground truth annotation is determined; the ground truth annotation includes the behavior label and the bounding box annotation. The upper bound of the localization quality is determined based on the largest bounding box complete intersection-union ratio in the matching result set related to the real annotation. The normalized denominator is determined based on the maximum comprehensive matching score in the set of matching results. Based on the comprehensive matching score, the upper bound of the positioning quality, and the normalized denominator, the target probability of the true annotation is adaptively adjusted to obtain the adjusted target probability.

7. A device for quantifying insect behavior by fusing spatial semantic features and spatiotemporal features, characterized in that, include: The acquisition module is used to acquire insect monitoring videos; The processing module is used to input the insect monitoring video into a multi-behavior recognition model for processing to obtain insect multi-behavior recognition results; the multi-behavior recognition model is used to extract local spatial features from video frames and spatiotemporal features from video segments, and to perform multi-behavior recognition based on the local spatial features and the spatiotemporal features; the insect multi-behavior recognition results include the time period of occurrence of each of the various behaviors of the insect; The processing module is also used to determine the insect's behavioral rhythm based on the insect's multi-behavior recognition results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the insect behavior quantification method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the insect behavior quantification method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the insect behavior quantification method as described in any one of claims 1 to 6.

Images