Target detection algorithm optimization method based on multi-modal fusion and related equipment

Abstract

A target detection algorithm optimization method based on multi-modal fusion and related equipment. The method acquires multi-modal soil pest situation data, and realizes the optimization of the target detection algorithm through operations such as feature processing, mapping, and channel setting. In the method, multi-modal soil pest situation data is acquired and time sequence features are extracted, which can comprehensively capture soil and pest situation information at different time points. The feature inversion matrix is generated by comparing the reversed time sequence feature sequence, which can mine potential laws; the feature difference degree between modes is calculated and a mode complementary graph is established to clarify the information supplement relationship; information transmission channels are established according to the weight and the features are transmitted to promote modal fusion. Finally, the information supplement ability is evaluated according to the feature similarity and the fusion weight is determined, which realizes dynamic adjustment of the mode weight, improves the adaptability and accuracy of the target detection algorithm to the complex farmland environment, reduces the feature redundancy and information loss, and enhances the accuracy of pest monitoring.

Description

Technical Field

[0001] This application relates to the field of smart agriculture technology, and in particular to an optimization method for target detection algorithm based on multimodal fusion and related equipment. Background Technology

[0002] With the rapid development of agricultural modernization, intelligent monitoring and control of farmland pests and diseases have become increasingly important. Target detection algorithms play a key role in farmland pest monitoring, collecting multimodal data of the farmland environment through various sensors, including images, sound, temperature, and humidity information, to achieve real-time monitoring and early warning of farmland pests.

[0003] In related technologies, static weights are used to fuse features from different modalities. Specifically, a pre-trained feature extraction model processes different modal information such as images and sensor data, and then linearly combines the features of each modality based on fixed weights (such as equal weighting) set by human experience. Finally, the detection results are output through a classifier or regression model.

[0004] However, farmland environments are complex and changeable, making it difficult for fixed-weight fusion methods to adapt to these changes. When the farmland environment changes, the importance of different modalities also changes. Fixed-weight feature fusion methods cannot dynamically adjust the weight distribution between modalities, resulting in insufficient exploration of the correlations between modal information and potentially leading to feature redundancy or information loss. Feature redundancy introduces interference, and information loss leads to the loss of key features, limiting target detection performance and making it difficult to meet the needs of accurate monitoring. Summary of the Invention

[0005] This application provides a target detection algorithm optimization method and related equipment based on multimodal fusion, which is used to solve the problem of limited target detection algorithm performance when monitoring and warning of pests in farmland, and improves the accuracy of target detection in complex farmland environments.

[0006] Firstly, this application provides an optimization method for a target detection algorithm based on multimodal fusion, used for real-time monitoring and early warning of farmland pests. The method includes: acquiring multimodal soil pest data in the operating environment of the target detection algorithm, and extracting time-series features of each modality from the multimodal soil pest data to obtain a time-series feature sequence. The multimodal soil pest data includes pest monitoring images, insect chirping audio, and pest activity video. The time-series feature sequence contains soil conditions and pest characteristics at different time points. The time-series feature sequence is then inverted along the time dimension to obtain an inverted sequence. The inverted sequence is compared with the time-series feature sequence at corresponding positions to generate a feature inversion matrix. Based on the feature inversion matrix, the feature distribution difference between any two different modalities in the time-series feature sequence and the inverted time sequence is calculated to obtain the intermodal feature difference degree. Based on the feature difference degree, a feature inversion matrix is ​​constructed. A modality complementarity graph is constructed, where each node represents a modality, and directed edges between nodes represent the information supplementation direction between different modalities. The weight of each directed edge is determined by the feature difference. Based on the weight values ​​of the directed edges in the modality complementarity graph, information transmission channels are established between different modalities sequentially in descending order of weight value. Bidirectional feature transfer between different modalities is performed within these information transmission channels, and the feature fusion result of the terminating modality is compared with the feature information of the starting modality to obtain feature similarity. The starting modality is the modality at the beginning of the directed edge in the modality complementarity graph, and the terminating modality is the modality at the end of the directed edge in the modality complementarity graph. The information supplementation capability of each modality is evaluated based on the feature similarity. Based on the information supplementation capability, the fusion weight between each modality is determined, and the fusion weight is updated to the target detection algorithm configuration library to achieve optimization and update of the target detection algorithm.

[0007] By adopting the above technical solution, multimodal soil insect data is first acquired and temporal features are extracted to comprehensively capture soil and insect information at different time points. Then, the temporal feature sequences are inverted and compared to generate a feature inverted matrix to uncover potential data patterns. Next, the feature difference degree between modalities is calculated and a modal complementarity graph is established to clarify the information supplementation relationship between modalities. Then, information transmission channels are established according to weights and bidirectional feature transmission is performed to promote deep modal fusion. Finally, the information supplementation capability is evaluated based on feature similarity, and the modal fusion weights are dynamically adjusted and updated to the algorithm configuration library. When the algorithm runs again, it reads the latest weights to fuse multimodal features. This method solves the performance limitation problem caused by insufficient utilization of modal information in real-time monitoring of agricultural insect infestations due to the inability of fixed weights to adapt to environmental changes. The dynamic weight allocation mechanism enables the algorithm to accurately adapt to environmental changes, allowing each modality to fully leverage its advantages in different scenarios, effectively avoiding feature redundancy and information loss, and ultimately significantly improving the accuracy of target detection in complex agricultural environments.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, the acquisition of multimodal soil pest data in the operating environment of the target detection algorithm, and the extraction of time-series features of each modality in the multimodal soil pest data to obtain a time-series feature sequence, specifically includes: acquiring multimodal soil pest data in the operating environment of the target detection algorithm, and synchronizing and aligning the multimodal soil pest data according to timestamps to obtain time-aligned multimodal soil pest data, which includes pest monitoring images, insect chirping audio, and pest activity video; extracting data features of each modality in the time-aligned multimodal soil pest data based on the target detection algorithm; and generating a time-series feature sequence from the data features of each modality according to the collection time point.

[0009] By adopting the above technical solution, multimodal soil pest data are synchronized and aligned according to timestamps, ensuring the consistency of the data in the time dimension and laying a solid foundation for subsequent analysis. Based on target detection algorithms, features of each modality are extracted to accurately capture key pest information. Then, a time-series feature sequence is generated according to the collection time, facilitating the analysis of pest dynamic trends and providing conditions for further in-depth exploration of time-series feature patterns. This, in turn, improves the target detection algorithm's ability to perceive and respond to dynamic changes in pests.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, based on the weight values ​​of the directed edges in the modal complement graph, information transmission channels are sequentially established between different modalities in descending order of the weight values. Specifically, this includes: comparing the weight values ​​of the directed edges in the modal complement graph with a preset weight threshold to obtain target directed edges that are greater than the preset weight threshold; establishing candidate transmission channels between the starting and ending modes of the target directed edges; calculating the correlation coefficient between the starting and ending modes in the candidate transmission channels at the same time point based on a soil insect infestation history database; determining channels with correlation coefficients higher than a first preset correlation threshold as reliable channels; and sequentially establishing information transmission channels between different modalities in descending order of the reliable channel weight values, where the reliable channel weight value is the weight value of the corresponding directed edge.

[0011] By employing the above technical solution, the weight values ​​of directed edges in the modal complementarity graph are compared with preset thresholds to filter out valuable target directed edges. Candidate transmission channels are established and correlation coefficients are calculated to ensure channel reliability. Channels with high correlation coefficients are identified as reliable channels and sorted by weight, with high-weight channels being prioritized for establishment. This ensures that important information transmission paths are established first, improving the efficiency and quality of information fusion, and helping target detection algorithms to quickly and accurately obtain key pest information, thereby enhancing detection efficiency.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, calculating the correlation coefficient between the starting mode and the ending mode in the candidate transmission channel at the same time point based on the soil insect infestation history database specifically includes: extracting a first historical feature value sequence of the starting mode and a second historical feature value sequence of the ending mode at the same time point from the soil insect infestation history database; aligning the first historical feature value sequence and the second historical feature value sequence in time to form a one-to-one corresponding feature value pair; and calculating the correlation coefficient of the feature value pair.

[0013] By employing the above technical solution, modal historical feature value sequences at the same time point are extracted from a soil insect infestation history database, and the intrinsic correlations between modalities are mined using historical data. The sequences are aligned to form feature value pairs, and correlation coefficients are calculated to quantify the degree of modal correlation. This calculation method based on historical data scientifically verifies the reliability of candidate transmission channels, provides a basis for selecting effective channels, and makes the information transmission of target detection algorithms more accurate and effective.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, information transmission channels are established sequentially between different modalities according to the order of the reliable channel weight values ​​from largest to smallest. Specifically, this includes: selecting the starting and ending modalities with larger weight values ​​as the first target modal pair for information transmission, according to the order of the reliable channel weight values ​​from largest to smallest; extracting environmental factor data related to the first target modal pair at the same time point from the soil pest history database, generating an environmental factor feature set, which includes farmland area, pest aggregation degree, and crop growth period; if the correlation of the environmental factor is lower than a second preset correlation threshold, then correcting the weight value of the first target modal pair based on the correlation of the environmental factor to obtain the corrected weight value; reordering all weight values ​​according to the corrected weight value, and re-establishing the information transmission channels between each modality sequentially from largest to smallest according to the new sorting result.

[0015] By adopting the above technical solution, modality pairs with higher weight values ​​are prioritized as the first target modality pairs, ensuring that key information transmission paths are processed first and improving information fusion efficiency. Environmental factor data is extracted from the soil pest history database and a feature set is generated, incorporating actual environmental factors into the consideration, making information transmission more aligned with real-world scenarios. When the correlation of environmental factors is low, the weight values ​​are adjusted to avoid unreasonable information transmission path selection due to environmental interference. All weight values ​​are reordered and information transmission channels are reconstructed sequentially, dynamically adjusting the order of information interaction between modalities. This series of operations allows the construction of information transmission channels to fully integrate environmental factors and modality weights, improving the accuracy and effectiveness of information transmission and enhancing the reliability and accuracy of target detection algorithms in capturing pest information in complex and variable farmland environments.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, bidirectional feature transmission of the different modalities is performed in the information transmission channel, and the feature fusion result of the terminating modality is compared with the feature information of the starting modality to obtain feature similarity. Specifically, this includes: receiving feature information of the starting modality and the terminating modality in the information transmission channel, and obtaining historical environmental data from the soil insect history database whose similarity to the current environmental conditions is higher than a third preset correlation threshold. The environmental data includes temperature, humidity, light intensity, and soil pH. Based on the historical environmental data, a feature mapping function from the starting modality to the terminating modality is constructed. The feature mapping function is used to map the feature information of the starting modality to the terminating modality, and the feature information of the terminating modality is fused to obtain the fused mapping feature of the terminating modality. The fused mapping feature is transmitted to the starting modality and compared with the feature information of the starting modality to obtain feature similarity.

[0017] By adopting the above technical solution, modal feature information is received and similar historical environmental data is acquired. A precise feature mapping function is constructed using historical experience. This function enables information complementarity between modalities, and the fused features are compared back to obtain feature similarity, thus evaluating the information transmission effect. The feature transmission and similarity evaluation mechanism ensures effective fusion and accurate transmission of modal information, enabling the target detection algorithm to capture pest features more comprehensively and accurately, thereby improving detection accuracy.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, based on the information supplementation capability, the fusion weights between the modalities are determined, and the fusion weights are updated to the target detection algorithm configuration library to achieve the optimization and update of the target detection algorithm. Specifically, this includes: monitoring the data quality indicators of each modality, which include data integrity and data timeliness; calculating the fusion weights of each modality based on the weighted combination of the information supplementation capability and the data quality indicators; and updating the fusion weights to the target detection algorithm configuration library to achieve the optimization and update of the target detection algorithm.

[0019] By adopting the above technical solution, the data quality indicators of each modality are monitored, and the data status is grasped in real time. Based on the information supplementation capability and the weighted calculation of data quality indicators, the fusion weight is comprehensively considered, taking into account the value of modal information and data reliability, and updated to the algorithm configuration library for dynamic optimization. This calculation method allows the algorithm to make reasonable use of data from each modality, enhancing the accuracy and robustness of target detection and better addressing the challenges of complex farmland environments.

[0020] In a second aspect, this application provides a target detection algorithm optimization system based on multimodal fusion, including one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the target detection algorithm optimization system based on multimodal fusion to perform the method as described in the first aspect and any possible implementation thereof.

[0021] Thirdly, this application provides a computer-readable storage medium including instructions that, when executed on a multimodal fusion-based target detection algorithm optimization system, cause the multimodal fusion-based target detection algorithm optimization system to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, this application provides a computer program product that, when run on a multimodal fusion-based target detection algorithm optimization system, causes the multimodal fusion-based target detection algorithm optimization system to perform the method described in the first aspect and any possible implementation thereof.

[0023] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. By employing techniques such as multimodal data temporal feature processing, constructing modal complementarity graphs, and dynamically adjusting fusion weights, this method first extracts temporal features from multimodal data and performs reverse comparison to uncover potential patterns and clarify information complementarity relationships between modalities. Then, information transmission channels are established based on weights to achieve feature transfer. Finally, the information complementarity is evaluated based on feature similarity, and weights are dynamically adjusted. This effectively solves the problems of fixed weights being unable to adapt to environmental changes and prone to feature redundancy or information loss, thereby improving the accuracy of target detection in complex farmland environments.

[0024] 2. By employing techniques such as selecting directed edges of targets, calculating the correlation coefficient of candidate channels, and constructing information transmission channels according to weights, valuable directed edges of targets are first selected using preset thresholds to lock in potential information transmission paths. Next, the correlation coefficient of candidate channels is calculated to verify the reliability of the paths. Finally, channels are established according to weight from largest to smallest to ensure that important information transmission paths are prioritized. This effectively solves the problems of unreliable information transmission paths and low information fusion efficiency, thereby achieving the effect of ensuring priority transmission of important information and improving the quality of information fusion.

[0025] 3. By employing techniques that combine historical environmental data to construct feature mapping functions, feature transfer, and feature similarity comparison, a precise feature mapping function is first constructed based on historical environmental data to promote information complementarity between modalities. Then, the fused features are fed back for comparison to obtain feature similarity and evaluate the transfer effect. Through feature transfer and similarity evaluation, information transfer is continuously optimized. This effectively solves the problems of insufficient information fusion and inaccurate transfer between modalities, thereby achieving the effect of strengthening modal information complementarity and accurately capturing pest characteristics. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating an optimization method for a target detection algorithm based on multimodal fusion in an embodiment of this application. Figure 2 This is another flowchart illustrating the target detection algorithm optimization method based on multimodal fusion in the embodiments of this application; Figure 3 This is a schematic diagram of a physical device structure of a target detection algorithm optimization system based on multimodal fusion in the embodiments of this application. Detailed Implementation

[0027] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0028] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0029] For ease of understanding, the method provided in this implementation is described in process below. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating a target detection algorithm optimization method based on multimodal fusion in an embodiment of this application.

[0030] 101. Obtain multimodal soil pest data in the running environment of the target detection algorithm, and extract the time series features of each mode in the multimodal soil pest data to obtain a time series feature sequence. The multimodal soil pest data includes pest monitoring images, insect chirping audio, and pest activity video. The time series feature sequence contains soil conditions and pest characteristics at different time points.

[0031] The target detection algorithm is a computer algorithm used to identify and locate pest targets in farmland. It is the core program of the entire system for pest monitoring and early warning. The operating environment refers to the farmland scenario in which the target detection algorithm actually runs, encompassing environmental factors such as geographical location, climate conditions, and soil type. Multimodal soil pest data includes soil pest-related information in different forms such as images, audio, and video. For example, pest monitoring images can intuitively present the morphology and quantity of pests, insect chirping audio can reflect the activity of pests, and pest activity videos can dynamically display the behavior of pests. Time series features refer to data features arranged in chronological order. In this step, it extracts the soil state and pest characteristics reflected in the multimodal data at different time points, such as changes in soil moisture and increases or decreases in the number of pests at different times. The final time series feature sequence is an ordered set of these time series features.

[0032] The system first acquires multimodal soil pest data, including pest monitoring images, insect chirping audio, and pest activity videos. Then, it processes the data for each modality separately, extracting key features reflecting soil conditions and pest changes. For pest monitoring images, target detection and segmentation algorithms are applied to extract pest morphological and density distribution features. For insect chirping audio signals, target detection algorithms are applied, including time-frequency domain analysis and deep learning-based acoustic event detection, to extract acoustic feature vectors and pest species characteristics. For pest activity videos, target detection algorithms are applied, including real-time multi-target tracking and motion behavior analysis, to extract spatiotemporal and activity patterns of target pests. Finally, these features are arranged chronologically to obtain the temporal feature sequence corresponding to each modality. By acquiring the temporal series features of multimodal data, the dynamic changes of pests and the evolutionary characteristics of different modalities over time can be captured. This provides rich and temporally contextualized foundational data for subsequent analysis of the correlations between different modalities and the mining of data complementarity. Only based on such dynamic data can pest conditions be detected and predicted more accurately.

[0033] 102. Reverse the time-series feature sequence in the time dimension to obtain the reversed sequence.

[0034] The time dimension represents the attributes of data on the time axis; it's a dimension describing how data changes over time. Reversal refers to reversing the chronological order of data in a time-series feature sequence. Data originally arranged in chronological order becomes data in reverse chronological order. A reversed sequence is a new sequence obtained by reversing a time-series feature sequence along the time dimension.

[0035] The system arranges the temporal feature sequences of each modality according to the timeline, starting from the data at the last time point and proceeding forward to the data at the first time point, thus obtaining the corresponding reverse sequence. For example, if the temporal feature sequence of a certain modality records the insect infestation feature data at minute 1, minute 2, and minute 3 in chronological order, after reversal, the order of the data in the reverse sequence becomes minute 3, minute 2, and minute 1. The inversion operation of the temporal feature sequence aims to uncover the asymmetric features of data in forward and reverse temporal order by constructing a time dimension reference in reverse order of the original sequence. This reveals the irreversible dynamic changes in the development of agricultural pests, such as the differences in body size changes of pests from egg to adult in forward and reverse time order. Simultaneously, the inverted sequence provides a counterfactual benchmark for subsequent feature comparisons. By comparing it point-by-point with the original temporal feature sequence, the stability and intensity of change of features at each time point can be quantitatively evaluated, identifying directional or periodic features. The asynchronicity of different modal data in the temporal dimension can also be amplified through the inversion operation, enabling the algorithm to more sensitively capture cross-modal temporal correlation patterns and discover the complementarity of different modalities in the temporal dimension. For example, the difference in sensitivity of audio data in the early stages of a pest outbreak compared to the importance of image data in the later stages provides crucial data support for dynamically adjusting fusion weights, improving the algorithm's adaptability to complex agricultural environments, optimizing resource allocation, and enhancing pest early warning capabilities.

[0036] 103. Compare the corresponding features of the inverted sequence with the time series feature sequence to generate a feature inversion matrix.

[0037] Corresponding positions represent data at the same time index position in the time-series feature sequence and the reversed sequence. Feature comparison refers to comparing the feature data at corresponding positions in the two sequences and analyzing the differences between them. The feature inversion matrix is ​​a two-dimensional matrix, with its rows and columns corresponding to the time-series feature sequence and the reversed sequence, respectively. The elements in the matrix are used to represent the degree of difference in feature data at corresponding positions.

[0038] The system compares the feature data at corresponding positions of each modality's inverted sequence with its corresponding temporal feature sequence, using the same time index. For example, it compares the feature data at the first time point in the temporal feature sequence with the feature data at the first time point in the inverted sequence (actually the last time point of the original sequence), calculating the difference between them. In this way, the system compares the feature data at all corresponding positions in the two sequences and fills the resulting differences or correlations into a two-dimensional matrix according to their corresponding row and column positions, ultimately generating a feature inversion matrix. This matrix comprehensively reflects the comparison of feature data between the temporal feature sequence and its inverted sequence at each position. Generating the feature inversion matrix is ​​intended to present the feature differences between the temporal feature sequence and its inverted sequence at each time point in a structured and intuitive way. This matrix format facilitates subsequent mathematical calculations and analysis to quantify the changing characteristics and degree of difference of different modal data in the time dimension, thus providing concrete data support for calculating the feature differences between modalities.

[0039] 104. Based on the inverted feature matrix, calculate the feature distribution difference between any two different modalities in the time series feature sequence and the inverted time series, and obtain the feature difference degree between the modalities.

[0040] Feature distribution differences refer to the differences in the distribution patterns and statistical properties of feature data from different modalities in time-series feature sequences and inverted time series. The intermodal feature dissimilarity is a quantitative indicator used to measure the degree of difference between feature data from any two different modalities.

[0041] The system first extracts feature data of different modalities from the inverted feature matrix in both the temporal feature sequence and the inverted temporal sequence. Then, for each pair of different modalities, it calculates the difference in their feature distributions in both the temporal feature sequence and the inverted temporal sequence. For example, for image and audio modalities, the system analyzes the differences in their feature data at different time points in terms of distribution range, central tendency, and dispersion. To calculate this difference, the system can employ various statistical methods and distance metrics algorithms, such as KL divergence, Wasserstein distance, and Bach distance. Taking KL divergence as an example, it measures the difference between two probability distributions. The system converts the feature data of different modalities into probability distribution form and then calculates the KL divergence value between them; a larger value indicates a greater difference in the feature distributions of the two modalities. By performing this calculation on all different modal pairs, the feature difference degree between any two modalities is obtained. Furthermore, the system can also obtain a feature difference degree matrix between modalities, where each element represents the degree of feature difference between the corresponding two modalities. The degree of feature difference provides an important basis for the subsequent construction of modal complementarity graphs, helping the system to understand the information differences and complementarity potential between different modalities, thereby enabling more effective multimodal fusion.

[0042] 105. Based on the feature difference degree, establish a modality complementarity graph. Each node in the modality complementarity graph represents a modality, and the directed edges between the nodes represent the information supplementation direction between different modalities. The weight of the directed edges is determined by the feature difference degree.

[0043] Modal complementarity graphs are used to represent the information complementarity relationships between different modalities. Nodes are the basic elements in the graph structure, and each node corresponds to a modal data, such as image modality, audio modality, video modality, etc. Directed edges represent the direction of information flow and are used to indicate the information complementarity relationship from one modality to another. The weight of a directed edge is a numerical value, which is determined by the feature difference degree between the modalities calculated in step 104. The larger the weight, the greater the information complementarity potential between the two modalities.

[0044] The system first determines the number of nodes in the modality complementarity graph based on the number of modalities, with each node representing a specific modality. Then, it establishes directed edges between nodes based on the feature differences between modalities. For each pair of different modalities, the system compares their feature differences and determines the weight and direction of the directed edge based on the magnitude of the difference. Generally, if the feature differences between modality A and modality B are large, and some features of modality A can compensate for the deficiencies of modality B, then the system establishes a directed edge from node A to node B, using the feature difference as the weight of this edge. For example, the image modality might be advantageous in pest morphology recognition, while the audio modality is more advantageous in pest activity detection. Therefore, there might be a directed edge from the image modality to the audio modality, indicating that information from the image modality can compensate for certain deficiencies in the audio modality. In this way, the system constructs a complete modality complementarity graph, intuitively demonstrating the information complementarity relationships and potential between different modalities. This modality complementarity graph provides the structural foundation for subsequently establishing information transmission channels, guiding the system on how to effectively transmit and fuse information between different modalities.

[0045] 106. Based on the weight values ​​of the directed edges in the modal complement graph, establish information transmission channels between different modalities in descending order of weight values.

[0046] Information transmission channels refer to the paths established between different modalities for transmitting and exchanging feature information. Through these channels, feature information from different modalities can be fused.

[0047] The system first extracts all directed edges and their weights from the modal complementarity graph, and sorts these edges in descending order of weight. After sorting, the system processes each directed edge sequentially, establishing an information transfer channel for the corresponding modal pair. When establishing this channel, the system can select appropriate techniques and methods based on the characteristics of the modality and the information transfer requirements. For example, for image modalities with spatial features and audio modalities with time-series features, the system can use an attention mechanism to establish the information transfer channel between them. The attention mechanism allows the model to automatically focus on feature information relevant to the current task in different modalities, improving the efficiency and accuracy of information transfer. Specifically, the system assigns an attention weight to each modality, and determines which feature information to extract from one modality and transfer to another based on these weights. For directed edges with larger weights, the system prioritizes establishing the information transfer channel and invests more computational resources and parameters to optimize this channel, ensuring that the information complementarity between the two modalities is fully utilized. By establishing information transmission channels sequentially according to their weight values ​​from largest to smallest, the system ensures that modality pairs with the greatest potential for information supplementation are utilized first, thereby improving the efficiency and effectiveness of the entire multimodal fusion process. These information transmission channels provide the foundation for subsequent feature transmission and fusion, enabling the transfer of feature information from different modalities within the system, ultimately enhancing the performance of the object detection algorithm.

[0048] 107. In this information transmission channel, bidirectional feature transmission of different modes is carried out, and the feature fusion result of the termination mode is compared with the feature information of the starting mode to obtain the feature similarity. The starting mode is the mode at the starting end of the directed edge in the complementary graph of the mode, and the termination mode is the mode at the pointing end of the directed edge in the complementary graph of the mode.

[0049] Bidirectional feature transfer means that information is not only transferred from the starting mode to the ending mode, but also back from the ending mode to the starting mode, realizing bidirectional exchange and fusion of feature information. The starting and ending modes are the modes represented by the two endpoints of a directed edge in a modality complementarity graph. The feature fusion result refers to the new features obtained by the ending mode after receiving information from the starting mode through a certain fusion algorithm. Feature similarity is a quantitative metric used to measure the degree of similarity between the feature fusion result of the ending mode and the original feature information of the starting mode.

[0050] The system initiates a bidirectional feature transfer process for each established information transfer channel. Taking an information transfer channel composed of image and audio modalities as an example, the system first extracts feature vectors containing key information such as insect morphology and location from the image modality and transmits them to the audio modality through the information transfer channel. After receiving the audio modality, it uses fusion algorithms such as attention mechanisms and feature splicing to integrate the image features with its own recorded features such as insect chirping frequency and intensity to generate a feature fusion result. After the transfer is completed, the system transmits the feature fusion result of the terminating modality (audio modality) to the starting modality (image modality) and compares the feature fusion result with the original feature information. During the comparison process, measurement methods such as cosine similarity and Euclidean distance can be used. Taking cosine similarity as an example, the system converts the feature fusion result and the starting modality features into vector form respectively, and determines the feature similarity by calculating the cosine value of the angle between the two vectors. If the cosine value is close to 1, it indicates that the terminating mode, after fusing information from the initiating mode, can effectively retain and reflect the key features of the initiating mode, demonstrating good feature transfer. Conversely, a low cosine value suggests significant differences between the two modes, indicating insufficient information fusion between them. By performing feature transfer and similarity calculations on all information transfer channels, the system can comprehensively evaluate the effectiveness of information interaction and fusion between different modes, providing an accurate data foundation for subsequent evaluation of the information supplementation capabilities of each mode.

[0051] 108. Evaluate the information supplementation capability of each modality based on the feature similarity.

[0052] Information supplementation capability refers to the ability of a modality to provide effective supplementary information to other modalities when interacting with them. Its strength reflects the importance and value of the modality in the multimodal fusion process.

[0053] The system takes each modality as its center and comprehensively analyzes the feature similarity of all information transmission channels it participates in. For a given modality (such as a video modality), if the feature similarity in its information transmission channels with other modalities (such as image and audio modalities) is generally high, it indicates that the modality can effectively transmit its own feature information to other modalities during feature transmission, demonstrating a strong information supplementation capability. Conversely, if the feature similarity is low, it means that the modality cannot fully play its supplementary role in information interaction, indicating a weak information supplementation capability. To more accurately assess information supplementation capability, the system can also use methods such as weighted averaging and analytic hierarchy process (AHP) to calculate feature similarity based on the weights of the information transmission channels (i.e., the feature differences between modalities), obtaining a comprehensive information supplementation capability score for each modality. For example, for information transmission channels with higher weights, their feature similarity accounts for a larger proportion in the score calculation because these channels reflect a more significant information complementarity relationship between modalities. In this way, the system can quantify the information supplementation capability of each modality, clearly understand the role and contribution of each modality in multimodal fusion, and provide key basis for subsequent determination of modality fusion weights.

[0054] 109. Based on the information supplementation capability, determine the fusion weights between the modalities, and update the fusion weights to the target detection algorithm configuration library to achieve the optimization and update of the target detection algorithm.

[0055] Fusion weights refer to the weight coefficients assigned to each modality during multimodal data fusion, used to adjust the contribution ratio of different modal feature information in the final fusion result. An object detection algorithm configuration library is a database or file system that stores parameters and configuration information related to object detection algorithms.

[0056] The system assigns fusion weights to each modality according to a specific mapping rule, based on the strength of their information supplementation capabilities. Generally, modalities with stronger information supplementation capabilities are assigned higher fusion weights because they provide more valuable information during multimodal fusion and contribute more significantly to improving object detection accuracy; conversely, modalities with weaker information supplementation capabilities are assigned lower weights. For example, the system can use normalization to map the information supplementation capability scores of each modality to weight values ​​between 0 and 1, ensuring that the sum of all weight values ​​is 1. Assuming the information supplementation capability scores for the image, audio, and video modalities are 0.6, 0.3, and 0.1, respectively, after normalization, their corresponding fusion weights are 0.6, 0.3, and 0.1. After determining the fusion weights, the system updates these weight values ​​in the object detection algorithm configuration library, overwriting the original fixed weight configuration. When the object detection algorithm runs again, it reads the latest fusion weights from the configuration library and fuses the multimodal feature information according to the new weight ratio, thereby optimizing and updating the algorithm. By dynamically adjusting the modality fusion weights, the target detection algorithm can better adapt to the complex and ever-changing farmland environment, give full play to the advantages of each modality, effectively avoid feature redundancy or information loss, significantly improve the accuracy and reliability of farmland pest target detection, and meet the actual needs of precise monitoring and early warning in smart agriculture.

[0057] The target detection algorithm optimization method based on multimodal fusion in this application embodiment acquires multimodal soil pest data and extracts temporal features. Through feature inversion and comparative analysis, a modality complementarity graph is established, information transmission channels are dynamically constructed, and the modality information supplementation capability is evaluated. This enables adaptive adjustment of modality fusion weights according to environmental changes, allowing the target detection algorithm to better adapt to the complex and ever-changing farmland environment, fully leverage the advantages of each modality, effectively avoid feature redundancy or information loss, and significantly improve the accuracy of farmland pest target detection.

[0058] Based on the above, the following is a more detailed description of the process provided in this implementation. Please refer to [link / reference]. Figure 2 This is another flowchart illustrating the target detection algorithm optimization method based on multimodal fusion in the embodiments of this application.

[0059] 201. Obtain multimodal soil pest data in the running environment of the target detection algorithm, and extract the time series features of each mode in the multimodal soil pest data to obtain a time series feature sequence. The multimodal soil pest data includes pest monitoring images, insect chirping audio, and pest activity video. The time series feature sequence contains soil conditions and pest characteristics at different time points.

[0060] The method specifically includes steps 2011 to 2013, which are not shown in the figure.

[0061] 2011. Obtain multimodal soil pest data in the running environment of the target detection algorithm, and synchronize and align the multimodal soil pest data according to the timestamp to obtain time-aligned multimodal soil pest data, which includes pest monitoring images, insect chirping audio, and pest activity video.

[0062] The operating environment refers to the actual farmland scenario in which the target detection algorithm runs, encompassing environmental factors such as geographical location, climate conditions, and soil type. Multimodal soil pest data is used to represent soil pest-related information in different forms, including images, audio, and video. A timestamp is a marker identifying the data collection time, used to record the specific moment the data was generated. Synchronization alignment refers to adjusting data from different modalities to the same time reference, ensuring the consistency of the data's temporal order. Time-aligned multimodal soil pest data refers to a data set where different modalities correspond one-to-one in time after time synchronization processing.

[0063] When the target detection algorithm runs in a farmland scenario, the system acquires multimodal soil pest data in real time, including pest monitoring images, insect sound audio, and pest activity video. Since the data collection times of different types of sensors may vary slightly, the system needs to perform time synchronization processing on this data to ensure the accuracy of subsequent analysis. The system extracts the corresponding timestamp information from each data point and then arranges and adjusts the data of different modalities according to time order, using a unified time benchmark, so that different modalities of data at the same point in time can be accurately correlated. For example, pest monitoring images, insect sound audio clips, and pest activity video frames collected within the same second are grouped into multimodal data at the same time point, thus obtaining time-aligned multimodal soil pest data, laying the foundation for subsequent feature extraction and analysis.

[0064] 2012. Based on the target detection algorithm, extract the data features of each modality in the time-aligned multimodal soil insect data.

[0065] Target detection algorithms refer to computer algorithms used to identify and locate pest targets in farmland. Data features of each modality refer to key information extracted from different types of data that reflects soil conditions and pest changes, such as morphological features of pests extracted from images, acoustic features extracted from audio, and motion features extracted from video.

[0066] After acquiring time-aligned multimodal soil pest data, the system uses target detection algorithms to extract features from each modality. For pest monitoring images, the system employs image processing and computer vision techniques, such as target detection and segmentation algorithms, to extract morphological features (e.g., shape, color, texture), quantity features, and distribution characteristics in farmland. For insect chirping audio, the system uses signal processing and pattern recognition techniques, such as time-frequency domain analysis and deep learning-based acoustic event detection, to extract acoustic features such as frequency, intensity, and duration of chirping, as well as pest species characteristics. For pest activity videos, the system combines video analysis and motion recognition techniques, such as multi-target real-time tracking and motion behavior analysis, to extract spatiotemporal features and activity patterns of pests, such as movement trajectories, activity speed, and group behavior. Through these feature extraction operations, the system can extract valuable information for pest monitoring and early warning from the raw multimodal data.

[0067] 2013. Generate a time-series feature sequence based on the data features of each modality according to the acquisition time point.

[0068] The data features of each modality refer to the key information extracted from different types of data that can reflect changes in soil conditions and insect infestations; the collection time point refers to the specific moment when the data was actually collected; the time-series feature sequence refers to the ordered sequence formed by arranging the data features of each modality in chronological order, which is used to record the soil conditions and insect infestation characteristics at different time points.

[0069] After extracting the data features for each modality, the system arranges these features in chronological order according to the collection time point corresponding to each feature. The system generates a separate temporal feature sequence for each modality, where each element corresponds to a feature data point in time. For example, for the image modality, the system arranges the morphological and quantitative features of pests extracted at different times each day in chronological order to form a temporal feature sequence for the image modality; similarly, corresponding temporal feature sequences are generated for the audio and video modalities. In this way, the system can organize discrete feature data into a dataset with time-series characteristics, facilitating subsequent analysis of the changing trends and patterns of pests over time, providing a basis for in-depth exploration of the relationships between temporal features and for optimizing target detection algorithms.

[0070] 202. Reverse the time-series feature sequence along the time dimension to obtain the reversed sequence. (This step has been explained in 102 and will not be repeated here.) 203. Compare the features at corresponding positions of the inverted sequence with the time-series feature sequence to generate a feature inversion matrix. (This step has been explained in 103 and will not be repeated here.) 204. Based on the inverted feature matrix, calculate the feature distribution difference between any two different modalities in the time-series feature sequence and the inverted time series, and obtain the intermodal feature difference degree. (This step has been explained in 104 and will not be repeated here.) 205. Based on the feature difference, construct a modality complementarity graph. Each node in the modality complementarity graph represents a modality, and the directed edges between nodes represent the information supplementation directions between different modalities. The weight of the directed edges is determined by the feature difference. (This step has been explained in 105 and will not be repeated here.) 206. Compare the weight values ​​of the directed edges in the modal complement graph with a preset weight threshold to obtain the target directed edges that are greater than the preset weight threshold.

[0071] The weight of a directed edge measures the potential for information supplementation between two modalities and is determined by the difference in features between the modalities. The preset weight threshold is a pre-defined critical value used to filter valuable directed edges, which can be set based on actual needs and historical data experience. Target directed edges are those with weight values ​​greater than the preset weight threshold; these edges represent strong potential for information supplementation between modalities.

[0072] After constructing the modal complementarity graph, the system needs to filter out edges with practical significance from all directed edges to improve the efficiency and quality of subsequent information transmission. The system iterates through all directed edges in the modal complementarity graph, obtains the weight value of each edge, and compares it with a preset weight threshold. The preset weight threshold can be set based on the distribution of inter-modal feature differences in historical data, for example, as the average or median of all edge weight values. Only when the weight value of a directed edge is greater than this threshold is it identified as a target directed edge. Through this filtering step, the system can eliminate edges with less potential for information supplementation and focus on processing target directed edges that are more valuable for modal fusion.

[0073] 207. Establish candidate transit channels between the starting and ending modes of the directed edge of the target.

[0074] The starting modality is the modal node at the beginning of the directed edge of the target, and the ending modality is the modal node at the end of the directed edge of the target. The candidate propagation path refers to the path initially established between the starting and ending modalities for feature propagation.

[0075] For each selected target directed edge, the system establishes a candidate transit channel between its corresponding starting and ending modalities. This candidate transit channel forms the initial architecture for information transmission, providing possible paths for subsequent feature transmission. When establishing a candidate transit channel, the system records information such as the starting and ending modalities, as well as the corresponding directed edge weights. For example, if there is a target directed edge pointing from the image modality to the audio modality, the system will establish a candidate transit channel between the image and audio modalities. This channel will be used to subsequently verify the reliability of feature transmission between these two modalities. By establishing candidate transit channels for all target directed edges, the system constructs a potential information transmission network, laying the foundation for subsequent selection of reliable information transmission channels.

[0076] 208. Based on the historical soil insect infestation database, calculate the correlation coefficient between the starting mode and the ending mode in the candidate transmission channel at the same time point.

[0077] The method specifically includes steps 2081 to 2083, which are not shown in the figure.

[0078] 2081. Extract the first historical feature value sequence of the starting mode and the second historical feature value sequence of the ending mode from the soil insect history database at the same time point.

[0079] The soil pest history database is a database used to store historical monitoring data of agricultural pests, containing pest characteristic data and environmental data at different time points and for different modalities. The first historical feature value sequence refers to the sequence formed by arranging the feature values ​​of the initial modality within the same period of historical data in chronological order. The second historical feature value sequence refers to the sequence formed by arranging the feature values ​​of the terminating modality within the same period as the initial modality in historical data in chronological order.

[0080] The system accesses a historical soil insect database and extracts feature values ​​from the starting and ending modes of each candidate transmission channel at the same historical time point. Based on the time range corresponding to the candidate transmission channel, the system retrieves the feature values ​​of the starting mode at each time point within that time range from the database, forming a first historical feature value sequence; simultaneously, it retrieves the feature values ​​of the ending mode at the same time point, forming a second historical feature value sequence. For example, if a candidate transmission channel corresponds to the transmission of information from an image mode to an audio mode, the system extracts image mode feature values ​​(such as insect morphological characteristics) and audio mode feature values ​​(such as insect chirping frequency characteristics) at the same time each day within a certain period from the historical database, forming the first and second historical feature value sequences respectively, providing data support for subsequent analysis of the correlation between modes.

[0081] 2082. Align the first historical feature value sequence and the second historical feature value sequence in time to form a one-to-one corresponding feature value pair.

[0082] Time alignment refers to matching feature values ​​at the same time point in two sequences to ensure that the feature value pairs correspond to modal data at the same moment. A feature value pair is a pair consisting of feature values ​​at the same time point from the first historical feature value sequence and the second historical feature value sequence.

[0083] Because there may be slight time deviations in the acquisition of feature values ​​from different modes in historical data, or differences in the time precision of data recording, the system needs to perform time alignment processing on the first and second historical feature value sequences. The system uses a unified time benchmark to calibrate the timestamps in both sequences, ensuring that each feature value corresponds to an accurate acquisition time point. Then, according to chronological order, feature values ​​at the same time point in the two sequences are paired to form feature value pairs. For example, for feature value A with a timestamp of "June 1, 2025, 10:00" in the first historical feature value sequence, a feature value B with the same timestamp is found in the second historical feature value sequence, and (A,B) is taken as a feature value pair. Through this time alignment operation, the system ensures that the two feature values ​​in the feature value pair reflect the states of the starting and ending modes at the same moment, laying the foundation for accurate calculation of the correlation coefficient.

[0084] 2083. Calculate the correlation coefficient of this eigenvalue pair.

[0085] The correlation coefficient is a statistic used to measure the degree of linear correlation between two variables. It typically ranges from -1 to 1, and the larger the absolute value, the stronger the correlation.

[0086] The system can employ various correlation coefficient calculation methods, such as Pearson correlation coefficient and Spearman correlation coefficient. Taking Pearson correlation coefficient as an example, its calculation steps are as follows: First, calculate the average of the first historical eigenvalue and the average of the second historical eigenvalue in all eigenvalue pairs; then, for each eigenvalue pair, calculate the difference between the first historical eigenvalue and the average, and the difference between the second historical eigenvalue and the average; next, calculate the sum of the products of these differences and divide by the product of the standard deviations of the first and second historical eigenvalues, and the number of eigenvalue pairs. Through this calculation, the obtained Pearson correlation coefficient can reflect the degree of linear correlation between the eigenvalues ​​of two modes. For example, if the calculated correlation coefficient is 0.8, it indicates a strong positive correlation between the eigenvalues ​​of the initial mode and the final mode; if the correlation coefficient is -0.5, it indicates a certain negative correlation. By calculating the correlation coefficient of eigenvalue pairs, the system can objectively evaluate the correlation strength between two modes in candidate transmission channels, providing a key quantitative indicator for screening reliable information transmission channels.

[0087] 209. The channel whose correlation coefficient is higher than the first preset correlation threshold is determined as a reliable channel.

[0088] The first preset correlation threshold is a pre-set critical value used to judge the reliability of a channel. It can be determined based on the distribution of correlation coefficients between modes in historical data and actual application requirements. A reliable channel refers to a candidate transmission channel whose correlation coefficient is higher than the first preset correlation threshold. These channels indicate a strong correlation between the initial mode and the final mode.

[0089] The system compares the correlation coefficient of each candidate transmission channel with a first preset correlation threshold. This threshold can be set to 0.5, 0.6, etc., with the specific value determined based on the actual scenario and historical data of farmland pest monitoring. Candidate transmission channels with correlation coefficients higher than this threshold are considered reliable channels. These channels indicate a strong correlation between the starting and ending modes in historical data, suggesting high reliability and effectiveness in information transmission. Candidate transmission channels with correlation coefficients lower than this threshold are considered unreliable and are not included in the subsequent information transmission channel construction process. Through this screening step, the system can identify truly reliable information transmission paths from the candidate channels, excluding channels with weak intermodal correlations, thereby ensuring the quality and efficiency of subsequent information transmission and fusion.

[0090] 210. Establish information transmission channels between different modalities in descending order of the reliable channel weight value. The reliable channel weight value is the weight value of the corresponding directed edge.

[0091] The method specifically includes steps 2101 to 2104, which are not shown in the figure.

[0092] 2101. According to the order of the reliable channel weight values ​​from largest to smallest, the starting mode and the ending mode with the larger weight values ​​are selected as the first target mode pair for information transmission.

[0093] The first target mode pair refers to the combination of the starting mode and the ending mode with the highest weight value selected from reliable channels, which serves as the primary processing object for information transmission.

[0094] After identifying reliable channels, the system sorts them by weight value to determine the priority of information transmission. The system extracts the weight values ​​of all reliable channels and sorts them from largest to smallest; a larger weight value indicates a stronger potential for inter-modal information supplementation. After sorting, the system selects the starting and ending modal corresponding to the reliable channel with the largest weight value as the first target modal pair. For example, if the reliable channel with the largest weight value is the image-to-video modal channel (weight value 0.85), then the image and video modal pairs are identified as the first target modal pair, and the construction of information transmission channels for this modal pair is prioritized to ensure that high-value information transmission paths are processed first.

[0095] 2102. From the soil pest history database, extract the environmental factor data related to the first target modality pair at the same time point, and generate an environmental factor feature set, which includes farmland area, pest aggregation degree and crop growth period.

[0096] Environmental factor data refers to data on environmental factors that influence changes in pest infestations. An environmental factor feature set is a dataset composed of environmental factors such as farmland area, pest aggregation level, and crop growth stage, used to analyze the impact of the environment on modal information transmission.

[0097] To ensure that the information transmission channels conform to the actual farmland environment, the system needs to combine environmental factors to evaluate the weight of modal pairs. The system accesses the historical soil pest database and extracts environmental factor data from the same historical time point for the first target modal pair (such as image modality and video modality). For example, it extracts data on farmland area (e.g., 10 mu, 15 mu), pest concentration (e.g., high density, medium density), and crop growth stage (e.g., seedling stage, maturity stage) corresponding to the feature collection time points of this modal pair in different months of 2024. After organizing this data by time point, an environmental factor feature set is generated, providing data support for subsequent analysis of the correlation between environmental factors and modal features.

[0098] 2103. If the correlation of the environmental factor is lower than the second preset correlation threshold, the weight value of the first target mode pair is corrected based on the correlation of the environmental factor to obtain the corrected weight value.

[0099] Environmental factor correlation refers to the degree of association between the environmental factor feature set and the feature data of the first target mode pair, which can be calculated using methods such as correlation coefficient. The second preset correlation threshold is a pre-set critical value for judging the degree of influence of environmental factors. If the environmental factor correlation is lower than this threshold, it indicates that the environment has a significant impact on modal information transmission, and the weights need to be adjusted. The adjusted weight values ​​are based on the original weight values, combined with the weights adjusted according to the environmental factor correlation, and are used to more accurately reflect the actual value of information transmission between modes.

[0100] The system calculates the correlation between the environmental factor feature set and the feature data of the first target modality pair (e.g., using the Pearson correlation coefficient to calculate the correlation between farmland area and the pest quantity feature in the image modality). If the calculated correlation is lower than the second preset correlation threshold (e.g., 0.4), it indicates that the current environmental factors (e.g., changes in farmland area) have a significant impact on the information transmission of the modality pair, and the original weight values ​​do not fully consider environmental factors, requiring correction. The correction method can be as follows: adjust the original weight values ​​proportionally according to the difference between the environmental factor correlation and the threshold (e.g., if the original weight is 0.85, the correlation is 0.3, the threshold is 0.4, and the difference is 0.1, then the corrected weight is 0.85 × (1 - 0.1 / 0.4) = 0.6375), making the weight values ​​more consistent with the modal complementarity in the actual environment.

[0101] 2104. Reorder all weight values ​​according to the revised weight values, and re-establish the information transmission channels between each modality in descending order according to the new sorting results.

[0102] Reordering refers to rearranging the weights of all reliable channels according to the corrected weight values; the new ordering result is the arrangement of the corrected weight values ​​from largest to smallest; re-establishing information transmission channels means constructing information transmission paths between each modality in sequence according to the new weight order.

[0103] The system substitutes the corrected weight values ​​of the first target modality pair into the weight set of all reliable channels and reorders all weight values. For example, if the original reliable channel weight order was: image to video (0.85), audio to image (0.78), video to audio (0.65), and the corrected weight of the first target modality pair was 0.6375, the new order would be: audio to image (0.78), video to audio (0.65), and image to video (0.6375). After sorting, the system establishes information transmission channels between modalities in the new order, prioritizing the construction of channels for modality pairs with larger weight values ​​(such as audio to image). This ensures that modality pairs with stronger information supplementation potential in the current environment are prioritized for feature transmission, thereby dynamically adjusting the information transmission path and improving the algorithm's adaptability to complex farmland environments.

[0104] 211. Receive the feature information of the starting mode and the ending mode in the information transmission channel, and obtain historical environmental data from the soil insect history database that has a similarity to the current environmental conditions that is higher than the third preset correlation threshold. The environmental data includes temperature, humidity, light intensity and soil pH.

[0105] Current environmental conditions refer to the real-time environmental parameters during the execution of the target detection algorithm. Historical environmental data refers to data collected in the past regarding temperature, humidity, light intensity, and soil pH, which are similar to the current environment. The third preset correlation threshold is a critical value used to judge the similarity between historical environmental data and the current environment, and can be set according to actual needs; historical environmental data with a similarity higher than the third preset correlation threshold indicates historical environmental data with a high degree of matching with the current environmental conditions.

[0106] The system receives feature information from the starting and ending modes in the channel, such as pest morphological features extracted from the image mode and insect chirping frequency features extracted from the audio mode. Simultaneously, the system retrieves historical environmental data from the soil insect infestation history database that has a similarity higher than a third preset correlation threshold (e.g., 0.7) to the current environmental conditions (e.g., current temperature 25℃, humidity 60%, light intensity 30000 lux, soil pH 6.5). The system calculates the similarity between the historical environmental data and the current environmental conditions, commonly using methods such as Euclidean distance or cosine similarity. For example, if a historical environmental data point has a temperature of 24℃, humidity of 58%, light intensity of 29000 lux, and soil pH of 6.3, its Euclidean distance to the current environment is calculated. If the distance is less than a set threshold (corresponding to a similarity higher than 0.7), the historical environmental data is included in the filtering results, forming a historical environmental dataset similar to the current environment, providing a reference for subsequent feature mapping function construction.

[0107] 212. Based on the historical environmental data, construct the feature mapping function from the starting mode to the ending mode.

[0108] Feature mapping functions are mathematical functions used to map initial modal feature information to the final modal feature space. They are constructed by the correlation between historical environmental data and corresponding modal features. The initial modality-to-final modality feature mapping represents the transformation of the initial modality features into a feature form compatible with the final modality, thus promoting modality fusion.

[0109] The system utilizes selected historical environmental data and their corresponding initial and final modal features to construct a feature mapping function. For example, for each sample in the historical environmental data (such as image modal feature A and audio modal feature B at temperature 24℃ and humidity 58%), the system analyzes the correspondence between the initial modal feature (feature A) and the final modal feature (feature B) in that environment. It trains a model using machine learning algorithms (such as linear regression, random forest, or neural networks) to learn the mapping pattern from the initial modal feature to the final modal feature. Taking linear regression as an example, the system assumes the feature mapping relationship is B = w × A + b, and fits parameters w and b using historical data to minimize the error between the mapped feature and the actual final modal feature. The resulting feature mapping function can predict the corresponding final modal feature based on the initial modal feature in the current environment, achieving feature conversion and transfer between modalities.

[0110] 213. Using the feature mapping function, the feature information of the starting mode is mapped to the ending mode, and then fused with the feature information of the ending mode to obtain the fused mapping feature of the ending mode.

[0111] The feature information of the initial modality refers to the insect infestation features extracted from the initial modality data. Mapping to the terminating modality means converting the initial modality features into a representation that conforms to the feature space of the terminating modality. The feature information of the terminating modality refers to the insect infestation features extracted from the terminating modality itself. Fusion means combining the mapped initial modality features with the terminating modality features to generate new features. The fused mapping features of the terminating modality refer to the feature set after fusing the initial modality features and the terminating modality features.

[0112] The system calls a constructed feature mapping function, inputting the feature information of the initial modality into the function to obtain features mapped to the feature space of the terminating modality. For example, inputting the pest morphology features of the image modality into the mapping function yields the corresponding representation of the audio modality feature space. Then, this mapped feature is fused with the feature information (such as insect chirping frequency features) of the terminating modality (e.g., the audio modality). Fusion methods include feature concatenation, weighted summation, or attention mechanisms. Weighted summation can be used; the system sets weights based on the information complementarity between modalities, such as a mapping feature weight of 0.6 and a terminating modality feature weight of 0.4, calculating the fused mapping feature: Fusion Feature = 0.6 × Mapping Feature + 0.4 × Audio Modality Feature. In this way, the fused mapping feature of the terminating modality can integrate supplementary information from the initial modality, enhancing the ability to represent insect infestations.

[0113] 214. Pass the fused mapping feature to the starting modality and compare it with the feature information of the starting modality to obtain the feature similarity.

[0114] Alignment refers to comparing the fused mapping features with the initial modality features and analyzing their similarity. Feature similarity is an indicator used to quantify the similarity between the fused mapping features and the initial modality features.

[0115] The system transmits the fusion mapping features of the terminating modality back to the starting modality via an information transmission channel; for example, it transmits the fusion mapping features of the audio modality back to the image modality. Then, it compares these fusion mapping features with the original feature information of the starting modality. Taking cosine similarity as an example, the system converts the fusion mapping features and the starting modality features into vector form and calculates the cosine of the angle between them: Similarity = (fusion feature vector × starting modality feature vector) / (|fusion feature vector| × |starting modality feature vector|). If the cosine value is close to 1, it indicates that the fusion mapping features have well preserved the key information of the starting modality, and the feature transmission effect is ideal; if the cosine value is low, it indicates that there is a deviation in the information fusion between modalities, and the mapping function or fusion strategy needs to be adjusted. Through the calculation of feature similarity, the system can evaluate the effectiveness of information transmission between modalities, providing data support for subsequent optimization of fusion weights.

[0116] 215. Evaluate the information supplementation capability of each modality based on the feature similarity. (This step has been explained in 108 and will not be repeated here.) 216. Based on the information supplementation capability, determine the fusion weights between the modalities, and update the fusion weights to the target detection algorithm configuration library to achieve the optimization and update of the target detection algorithm.

[0117] The method specifically includes steps 2161 to 2163, which are not shown in the figure.

[0118] 2161. Monitor the data quality indicators for each modality, including data integrity and data timeliness.

[0119] Data quality metrics are standards used to evaluate the reliability of data across various modalities, including data integrity and data timeliness. Data integrity refers to whether there is any loss or corruption in the data during collection, transmission, and storage, such as whether images are blurry or audio is interrupted. Data timeliness refers to whether the data can reflect the current insect infestation situation in a timely manner, such as whether video frames are real-time acquisitions rather than historical data.

[0120] The system monitors the acquisition and transmission of data from each modality in real time. For pest monitoring images, the system checks the image resolution, clarity, and whether there are any missing areas. If half of an image is black due to sensor malfunction at a certain moment, the image data is considered to have low integrity. For insect chirping audio, the system checks the continuity of the audio signal. If there is a silent segment exceeding 10 seconds, the data is considered incomplete. Regarding data timeliness, the system compares the data acquisition timestamp with the current time. If the timestamp of video data is 30 minutes ago, while the system's timeliness threshold is 15 minutes, the video data is considered insufficiently timely. By continuously monitoring these indicators, the system can grasp the quality status of each modality of data in real time.

[0121] 2162. Based on the weighted combination of the information supplementation capability and the data quality index, calculate the fusion weight of each modality.

[0122] Weighted combination refers to assigning different weight coefficients to information supplementation capabilities and data quality indicators to comprehensively evaluate the importance of modalities; fusion weight is the proportion of each modality in the feature fusion process, and the larger the value, the greater the contribution of that modality to the fusion result.

[0123] The system first standardizes information supplementation capability and data quality indicators, placing them within the same numerical range (e.g., 0-1). For example, if a modality has an information supplementation capability score of 0.7, a data integrity score of 0.8, and a data timeliness score of 0.6, then the overall data quality indicator score is (0.8 + 0.6) / 2 = 0.7. Then, based on the actual needs of farmland monitoring, the system assigns weight coefficients to information supplementation capability and data quality indicators, such as a weight of 0.6 for information supplementation capability and 0.4 for data quality indicators. At this point, the fusion weight for this modality is calculated as: 0.7 × 0.6 + 0.7 × 0.4 = 0.7. If a modality has low data integrity (e.g., a score of 0.4), even if its information supplementation capability score is high (e.g., 0.8), the final fusion weight will decrease due to the drag from the data quality indicator, resulting in 0.8 × 0.6 + 0.4 × 0.4 = 0.64, ensuring that modalities with poor data quality do not excessively affect the fusion results.

[0124] 2163. Update the fusion weight to the target detection algorithm configuration library to achieve the optimization and update of the target detection algorithm.

[0125] The object detection algorithm configuration library is a database that stores the algorithm's runtime parameters, including the fusion weights for each modality. Optimization and updating refers to replacing the old weights in the configuration library with the newly calculated fusion weights, enabling the algorithm to adjust its modality fusion strategy based on real-time data.

[0126] The system packages the calculated fusion weights for each modality (e.g., 0.6 for image modality, 0.3 for audio modality, and 0.1 for video modality) into a configuration file and writes it to the object detection algorithm configuration library via an interface. When the algorithm runs again, it reads the latest fusion weights from the configuration library. For example, during the feature fusion stage, image features are multiplied by 0.6, audio features by 0.3, and video features by 0.1 before being weighted and summed. If, at some point, the timeliness of video data drops sharply due to equipment failure (e.g., the timeliness score drops from 0.8 to 0.2), the system will recalculate the fusion weights, and the weight of the video modality will be reduced accordingly (e.g., from 0.1 to 0.05). The algorithm will reduce its reliance on outdated video data and instead utilize more effective data from the image and audio modalities, thereby improving detection accuracy in complex environments and achieving dynamic optimization of the algorithm.

[0127] The target detection algorithm optimization method based on multimodal fusion in this application embodiment acquires multimodal soil pest data and extracts temporal features. A feature inversion matrix is ​​generated through temporal reversal comparison. The feature difference degree between modalities is calculated, and a modal complementarity graph is constructed. Information transmission channels are dynamically filtered and corrected. Historical environmental data is combined to achieve feature transmission and fusion between modalities. Finally, based on information supplementation capability and data quality indicators, the fusion weight is calculated and updated to the algorithm configuration library. This achieves dynamic optimization of the target detection algorithm in complex farmland environments. It not only solves the problems of traditional fixed-weight fusion methods being unable to adapt to environmental changes and prone to feature redundancy or information loss, but also significantly improves the accuracy and reliability of farmland pest monitoring, effectively explores the correlation between different modal information, and enhances the accuracy of pest monitoring.

[0128] The following describes the target detection algorithm optimization system based on multimodal fusion in the embodiments of this invention from the perspective of hardware processing. Please refer to [link / reference]. Figure 3 This is a schematic diagram of a physical device structure of a target detection algorithm optimization system based on multimodal fusion in an embodiment of this application.

[0129] It should be noted that, Figure 3 The structure of the target detection algorithm optimization system based on multimodal fusion shown is only an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0130] like Figure 3As shown, the target detection algorithm optimization system based on multimodal fusion includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in Read-Only Memory (ROM) 302 or a program loaded from storage section 308 into Random Access Memory (RAM) 303, such as executing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0131] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0132] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the various functions defined in the present invention.

[0133] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0135] Specifically, the target detection algorithm optimization system based on multimodal fusion in this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the target detection algorithm optimization method based on multimodal fusion provided in the above embodiment.

[0136] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the multimodal fusion-based target detection algorithm optimization system described in the above embodiments; or it may exist independently and not assembled into the multimodal fusion-based target detection algorithm optimization system. The storage medium carries one or more computer programs, which, when executed by a processor of the multimodal fusion-based target detection algorithm optimization system, cause the multimodal fusion-based target detection algorithm optimization system to implement the multimodal fusion-based target detection algorithm optimization method provided in the above embodiments.

[0137] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0138] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0139] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0140] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An optimization method for a target detection algorithm based on multimodal fusion, characterized in that, The method for real-time monitoring and early warning of agricultural pests includes: The target detection algorithm obtains multimodal soil insect data in the operating environment, and extracts the time series features of each mode in the multimodal soil insect data to obtain a time series feature sequence. The multimodal soil insect data includes insect monitoring images, insect chirping audio, and insect activity video. The time series feature sequence contains soil conditions and insect characteristics at different time points. The time-series feature sequence is inverted along the time dimension to obtain the inverted sequence; The inverted sequence is compared with the temporal feature sequence at corresponding positions to generate a feature inversion matrix; Based on the inverted feature matrix, the feature distribution difference between any two different modalities in the temporal feature sequence and the inverted sequence is calculated to obtain the intermodal feature difference degree. A modal complementarity graph is established based on the feature difference degree. Each node in the modal complementarity graph represents a modality, and the directed edges between the nodes represent the information supplementation direction between different modalities. The weight of the directed edges is determined by the feature difference degree. Based on the weight values ​​of the directed edges in the modal complementarity graph, information transmission channels are established between different modalities in descending order of the weight values. In the information transmission channel, bidirectional feature transmission of the different modalities is carried out, and the feature fusion result of the terminating modality is compared with the feature information of the starting modality to obtain the feature similarity. The starting modality is the modality at the starting end of the directed edge in the modality complement graph, and the terminating modality is the modality at the pointing end of the directed edge in the modality complement graph. The information supplementation capability of each modality is evaluated based on the aforementioned feature similarity. Based on the information supplementation capability, the fusion weights between the modalities are determined, and the fusion weights are updated to the target detection algorithm configuration library to achieve the optimization and update of the target detection algorithm.

2. The method according to claim 1, characterized in that, The acquisition of multimodal soil insect data in the operating environment of the target detection algorithm, and the extraction of time-series features for each modality in the multimodal soil insect data to obtain a time-series feature sequence, specifically includes: The target detection algorithm acquires multimodal soil pest data in the operating environment, and synchronizes and aligns the multimodal soil pest data according to timestamps to obtain time-aligned multimodal soil pest data, which includes pest monitoring images, insect chirping audio, and pest activity video. Based on the target detection algorithm, data features of each modality in the time-aligned multimodal soil insect data are extracted; The data features of each modality are used to generate a time-series feature sequence according to the acquisition time point.

3. The method according to claim 1, characterized in that, Based on the weight values ​​of the directed edges in the modal complementarity graph, information transmission channels are established sequentially between different modalities in descending order of weight values, specifically including: The weight values ​​of the directed edges in the modal complement graph are compared with a preset weight threshold to obtain target directed edges that are greater than the preset weight threshold. Establish a candidate propagation channel between the starting mode and the ending mode of the target directed edge; Based on the historical soil insect infestation database, the correlation coefficient between the initial mode and the termination mode in the candidate transmission channel at the same time point is calculated. Channels with correlation coefficients higher than a first preset correlation threshold are identified as reliable channels; Information transmission channels are established sequentially between different modalities according to the reliable channel weight values ​​in descending order, where the reliable channel weight value is the weight value of the corresponding directed edge.

4. The method according to claim 3, characterized in that, The calculation of the correlation coefficient between the initial mode and the final mode in the candidate transmission channel at the same time point, based on the historical soil insect infestation database, specifically includes: Extract the first historical feature value sequence of the starting mode and the second historical feature value sequence of the ending mode at the same time point from the soil insect infestation history database; The first historical feature value sequence and the second historical feature value sequence are time-aligned to form a one-to-one corresponding feature value pair; Calculate the correlation coefficient of the eigenvalue pairs.

5. The method according to claim 3, characterized in that, Information transmission channels are established sequentially between different modalities according to the reliable channel weight values ​​in descending order, specifically including: According to the reliable channel weight values ​​in descending order, the starting mode and the ending mode with the larger weight values ​​are selected first as the first target mode pair for information transmission. From the soil pest history database, extract environmental factor data related to the first target modality pair at the same point in time to generate an environmental factor feature set, which includes farmland area, pest aggregation degree and crop growth period; If the correlation of the environmental factors is lower than the second preset correlation threshold, the weight value of the first target mode pair is corrected based on the correlation of the environmental factors to obtain the corrected weight value. All weight values ​​are reordered according to the revised weight values, and the information transmission channels between each modality are re-established in descending order of the new sorting results.

6. The method according to claim 3, characterized in that, In the information transmission channel, bidirectional feature transmission of the different modalities is performed, and the feature fusion result of the terminating modality is compared with the feature information of the starting modality to obtain feature similarity, specifically including: The system receives feature information of the starting mode and the ending mode in the information transmission channel, and obtains historical environmental data from the soil insect infestation history database that has a similarity to the current environmental conditions that is higher than a third preset correlation threshold. The environmental data includes temperature, humidity, light intensity and soil pH. Based on the historical environmental data, a feature mapping function from the starting mode to the ending mode is constructed; The feature mapping function is used to map the feature information of the starting mode to the ending mode, and then the feature information of the ending mode is fused to obtain the fused mapping feature of the ending mode; The fused mapping features are passed to the starting modality and compared with the feature information of the starting modality to obtain feature similarity.

7. The method according to claim 1, characterized in that, Based on the information supplementation capability, the fusion weights between the modalities are determined, and the fusion weights are updated to the target detection algorithm configuration library to optimize and update the target detection algorithm. Specifically, this includes: Monitor data quality indicators for each modality, including data integrity and data timeliness; Based on the weighted combination of the information supplementation capability and the data quality indicators, the fusion weight of each modality is calculated; The fusion weights are updated in the target detection algorithm configuration library to optimize and update the target detection algorithm.

8. A target detection algorithm optimization system based on multimodal fusion, characterized in that, Includes one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, and the one or more processors calling the computer instructions to cause the multimodal fusion-based target detection algorithm optimization system to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are run on the multimodal fusion-based target detection algorithm optimization system, the multimodal fusion-based target detection algorithm optimization system performs the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on the multimodal fusion-based target detection algorithm optimization system, the multimodal fusion-based target detection algorithm optimization system performs the method as described in any one of claims 1-7.

Images