A crop pest intelligent identification and early warning system based on a multi-modal large language model

Abstract

The application relates to the technical field, in particular to a crop pest intelligent identification and early warning system based on a multimodal large language model.In the application, based on a numerical difference and fluctuation detection mechanism, mutations and abnormal sampling are excluded, time correlation characteristics of parameters such as temperature and humidity, illumination and wind speed are extracted, the occurrence environment of pests is effectively restored, after image and environmental data are aligned through time indexing, a unified time sequence feature structure is constructed through transverse splicing, linear scaling and missing processing, a single-channel fusion representation vector is constructed by introducing a generative adversarial network for weighted combination, the consistency and representation accuracy of cross-modal data are improved, the nearest neighbor class center is matched by using the Euclidean distance, and the difference superposition and center updating of the deviated vector are performed in combination with the isolated forest, so that the label result has self-adaptive expansion capability and abnormal recognition capability, and is matched with a preset message template, the push content is formatted into a short message and an application message through interface calling, and immediate reporting of a pest event is realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent pest identification technology, and in particular to an intelligent identification and early warning system for crop pests based on a multimodal large language model. Background Technology

[0002] The field of intelligent pest identification technology focuses on using computer vision and artificial intelligence methods to automatically identify and diagnose crop diseases and pests, and combining model decision-making mechanisms to achieve pest and disease level assessment and early warning. This includes image recognition algorithms, classification models, pest knowledge bases, pathological atlas analysis and reasoning mechanisms, to achieve high precision, high efficiency, low latency in pest identification, and scalability.

[0003] The purpose of this intelligent identification and early warning system for crop pests based on a multimodal large language model is to build a highly automated pest identification and response mechanism, enabling joint analysis of image data, descriptive text information, and environmental context for different types of crop pests. This aims to improve the accuracy of pest identification and the timeliness of early warning response, and ultimately achieve the goals of crop yield protection and disease control.

[0004] Existing technologies generally rely on static image recognition models to process single-frame images. When the outline of insect spots is unclear or the background interference is strong, it is easy to produce missed or false judgments. It is difficult to meet the needs of insect pest identification in densely planted areas. Moreover, the coarse-grained acquisition method lacks modeling of continuous temporal features and a mechanism for removing outlier values, resulting in serious data noise interference and easy to cause incorrect judgments. In terms of multimodal data fusion, simple splicing or shallow connection operations are used, which are difficult to handle dimensional differences and nonlinear correlations, affecting the fusion expression effect and reducing the recognition accuracy of the classification model. At the same time, they generally rely on static label classification, which lacks flexibility when facing new or marginal insect pests. They cannot dynamically adjust the category center and label structure, which limits the system's ability to adapt to abnormal insect pests. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and propose an intelligent identification and early warning system for crop pests based on a multimodal large language model.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A smart identification and early warning system for crop pests based on a multimodal large language model includes:

[0007] The lesion capture module acquires a panoramic view of the plant through a camera, collects continuous frames of leaf surface images, calculates color differences and performs edge filtering, performs connected component judgment and region clustering screening, and obtains insect spot views.

[0008] Environmental data modeling module: Based on the insect spot view, collect temperature, humidity, light and wind speed data, filter out abnormal sections through difference calculation and fluctuation detection, extract time-related feature values, and form an environmental parameter matrix;

[0009] Multimodal fusion module: Based on the insect spot view and environmental parameter matrix, the image channel values ​​and environmental parameter values ​​are concatenated in chronological order. A generative adversarial network is used to perform numerical range scaling and dimension correction, complete scale unification and vector combination, and obtain the fusion description vector.

[0010] Intelligent classification module: Calculates the Euclidean distance from the fused description vector to the center vector of each category, selects matching labels according to the center index corresponding to the distance, and adopts isolated forest. If the distance exceeds the threshold, it extracts the difference vector between the current vector and the neighboring center, superimposes it on the original center coordinates and appends the label number to generate pest category labels.

[0011] Alarm scheduling module: Based on the pest category tags, it matches the level number with the notification template, fills the push text with formatted fields, calls the SMS and application interfaces and sends the message, collects the response and writes it to the log, and generates an alarm task list.

[0012] As a further aspect of the present invention, the lesion capture module includes:

[0013] Image acquisition submodule: Uses multi-angle cameras to acquire panoramic images of the plant and capture continuous frame images of the leaf surface. By completing image acquisition within a fixed time interval and constructing a frame sequence structure, and performing pixel matrix reading and timestamp recording, the frame image number and sampling batch identifier are completed to obtain a set of frame images;

[0014] Edge aggregation submodule: Based on the frame image set, it calculates the adjacent difference layer by extracting the gray value of each pixel in the image, filters out low-change areas and retains the edge response area, establishes a connected graph through the adjacent pixel index and generates a set of boundary blocks, and performs spatial clustering to merge similar areas in combination with the position coordinates to obtain the worm spot view.

[0015] As a further aspect of the present invention, the environmental data modeling module includes:

[0016] Parameter acquisition submodule: Based on the insect spot view, multiple sets of sensors are used to collect temperature, humidity, light and wind speed values. A multi-channel parameter stream is constructed through the time series corresponding to the sampling points. Data unit standard conversion and sampling label binding are performed and integrated into a unified structure record to obtain the original parameter set.

[0017] Feature construction submodule: Based on the original parameter set, identify mutation nodes by calculating the magnitude of numerical changes at continuous time points and perform fluctuation range positioning, set upper and lower thresholds to filter out outlier records and retain stable data segments, extract time labels for each parameter and construct structured sequences to obtain the environmental parameter matrix.

[0018] As a further aspect of the present invention, the multimodal fusion module includes:

[0019] Sequence splicing submodule: Based on the insect spot view and environmental parameter matrix, a unified time index is used to perform a one-to-one mapping between image channel values ​​and environmental parameter values. A multi-dimensional time-series vector is constructed by horizontal splicing and the source field labels are marked to obtain the time-series feature set.

[0020] Interval scaling submodule: Based on the time series feature set, it performs an interval linear transformation on the logarithmic values ​​of each column of valley and peak values, adjusts the numerical amplitude through normalization mapping, and simultaneously processes missing fields and abnormal nodes to obtain a correction vector set;

[0021] Vector merging submodule: Based on the correction vector set, a generative adversarial network is used to weight and combine each field to construct a single-channel fusion structure. The image and parameter feature vector are merged by field pairing and dimensionality unification processing is performed to obtain the fusion description vector.

[0022] As a further aspect of the present invention, the generative adversarial network first initializes the parameters of the generator network and the discriminator network, inputs the correction vector set into the generator, the generator sequentially performs multi-layer linear mapping and activation transformation, outputs a fused candidate vector, and then inputs the fused candidate vector and the real temporal feature vector samples into the discriminator. The discriminator calculates the discrimination scores of real and fake samples through the classification layer, then calculates the adversarial loss based on the discriminator output and the real label, performs gradient backpropagation and parameter updates on the discriminator and the generator respectively, and finally repeats the above generation and discrimination loop until the misclassification rate of the discriminator on the generated samples reaches a preset standard.

[0023] As a further aspect of the present invention, the generative adversarial network is defined according to the formula:

[0024]

[0025] Where: v′ f Let W represent the fusion description vector, σ represent the activation function, τ represent the temperature coefficient, and W represent the temperature coefficient. c Let v represent the fusion transformation weight matrix, α represent the fusion weights of image and parameter features, and v img The feature vector representing the crop pest image field, v param This represents the feature vector of environmental parameters associated with pests, γ represents the weighting coefficient of the auxiliary feature vector, and v auxThis represents the contextual auxiliary features generated by the multimodal large language model, where η represents the gating coefficient between the image and the parameter field, and b c δ represents the fusion bias term, and δ represents the bias correction term.

[0026] As a further aspect of the present invention, the intelligent classification module includes:

[0027] Distance calculation submodule: Based on the fused description vector, it performs item-by-item difference operation with the center vector of each category using the coordinate correspondence between vectors, calculates the Euclidean distance by accumulating the squared differences, and outputs the distance set to obtain the distance distribution;

[0028] Tag extraction submodule: Based on the distance distribution, the corresponding tag number is retrieved using the minimum distance value index, the category name is obtained by matching the tag mapping table and combined to form a standard classification identifier, and the initial tag is obtained;

[0029] The central update submodule: Based on the fused description vector and the initial label, it adopts an isolated forest. When the distance valley value is higher than the set threshold, it extracts the difference between the vector and the original central vector and performs coordinate accumulation. At the same time, it adds the corresponding number to the label sequence to obtain the pest category label.

[0030] As a further aspect of the present invention, the isolated forest first constructs multiple random subsample sets and independently trains binary split trees on each subsample set. During the tree construction process, feature fields and splitting thresholds are randomly selected for dividing samples. Then, for the input fused description vector, the path is traversed through all trees, and the path length to reach the leaf node in each tree is recorded. Next, the average path length in all trees is used as the evaluation basis. Based on the set anomaly threshold, it is determined whether the vector deviates from the current classification center. If the deviation condition is met, the difference between the vector and the center vector is extracted, the center position is updated, and a new label number is added to generate a pest category label.

[0031] As a further aspect of the present invention, the isolated forest is defined according to the formula:

[0032]

[0033] Among them: I i The improved distance index represents the distance between the fused description vector and the center of the i-th type of pest, where v represents the pest description vector obtained by fusing image, parameters, and linguistic semantics, and c represents the distance between the fused description vector and the center of the i-th type of pest. i This represents the current center vector of the i-th type of pest. Let α represent the inverse of the covariance matrix of the i-th type of pest sample. ′ β represents the coefficient of the structural correlation weighting term, P represents the spatial structural correlation matrix of crop pests, and β represents the coefficient of the structural correlation weighting term. ′γ represents the constraint coefficient of the auxiliary vector on the fusion result, z represents the auxiliary semantic vector output by the anomaly detection tree in the isolated forest model, and γ represents the constraint coefficient of the auxiliary vector on the fusion result. ′ θ represents the regularization constant, and θ represents the preset label update threshold bias term.

[0034] As a further aspect of the present invention, the alarm scheduling module includes:

[0035] Template filling submodule: Based on the pest category tags, the module retrieves the category number using a mapping table and extracts the corresponding message template. It then replaces the tag number and category name in the field and injects the template text. Finally, it concatenates message paragraphs and separators to construct a push list and obtain the push content set.

[0036] Push execution submodule: Based on the push content set, it calls the SMS interface to submit each text message and receive the status response by traversing the list, and calls the application interface to receive the confirmation information by traversing the same text message. It also writes the status response and confirmation information to the log and updates the task status, and obtains the alarm task list.

[0037] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0038] In this invention, based on the numerical difference and fluctuation detection mechanism, abrupt changes and abnormal sampling are excluded, and the temporal correlation features of parameters such as temperature, humidity, light, and wind speed are extracted to effectively restore the pest occurrence environment. After the image and environmental data are aligned by time index, a unified temporal feature structure is constructed through horizontal stitching, linear scaling and missing data processing. A generative adversarial network is introduced for weighted combination to construct a single-channel fusion representation vector, thereby improving the consistency and representation accuracy of cross-modal data.

[0039] In this invention, Euclidean distance is used to match the nearest neighbor category centers, and isolated forest is used to perform difference superposition and center update on the deviation vector, so that the label results have adaptive expansion capability and anomaly recognition capability. It also matches a preset message template and formats the pushed content into SMS and application messages through interface calls, so as to realize the instant notification of pest events, comprehensively improve the recognition efficiency and response time, and has deployability and versatility. Attached Figure Description

[0040] Figure 1 This is a system flowchart of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0042] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0043] Example 1

[0044] Please see Figure 1 This invention provides a technical solution: an intelligent identification and early warning system for crop pests based on a multimodal large language model, comprising:

[0045] The lesion capture module acquires a panoramic view of the plant through a camera, collects continuous frames of leaf surface images, calculates color differences and performs edge filtering, performs connected component judgment and region clustering screening, and obtains insect spot views.

[0046] Environmental data modeling module: Based on insect spot view, collect temperature, humidity and light and wind speed data, screen out abnormal sections through difference calculation and fluctuation detection, extract time-related feature values, and form an environmental parameter matrix;

[0047] Multimodal fusion module: Based on the insect spot view and environmental parameter matrix, the image channel values ​​and environmental parameter values ​​are concatenated in chronological order. A generative adversarial network is used to perform numerical range scaling and dimension correction, complete scale unification and vector combination, and obtain the fusion description vector.

[0048] Intelligent classification module: Calculates the Euclidean distance from the fused description vector to the center vector of each category, selects matching labels according to the center index corresponding to the distance, and adopts an isolated forest. If the value exceeds the threshold, it extracts the difference vector between the current vector and the neighboring center, superimposes it on the original center coordinates and appends the label number to generate pest category labels.

[0049] Alarm scheduling module: Based on pest category tags, it matches level numbers and notification templates, fills push text with formatted fields, calls SMS and application interfaces to send messages, collects responses and writes them to logs, and generates an alarm task list.

[0050] The lesion capture module includes:

[0051] Image acquisition submodule: Uses multi-angle cameras to acquire panoramic images of the plant and capture continuous frame images of the leaf surface. By completing image acquisition within a fixed time interval and constructing a frame sequence structure, and performing pixel matrix reading and timestamp recording, the frame image number and sampling batch identifier are completed to obtain a set of frame images;

[0052] Edge aggregation submodule: Based on the frame image set, it calculates the adjacent difference layer by extracting the gray value of each pixel in the image, filters out low change areas and retains the edge response area, establishes a connected graph through the adjacent pixel index and generates a set of boundary blocks, and performs spatial clustering to merge similar areas by combining the position coordinates to obtain the insect spot view;

[0053] Image acquisition submodule: Uses multi-angle cameras to acquire panoramic images of the plant and capture continuous frame images of the leaf surface. It controls the image acquisition process with a fixed time interval, calls the image acquisition command to set the frame rate to 5 frames per second, performs image acquisition caching operations and saves the image sequence as image frames, reads the content of each frame image and parses it into a three-dimensional pixel matrix structure, extracts the image acquisition time and records it as a second-level timestamp field, constructs the frame image naming rule by stitching the image frame number with the current timestamp, writes the image frame and the corresponding timestamp field into the sequence data table in the form of key-value pairs, uses a random generation command to generate sampling batch identifiers and bind them to the corresponding image frame set, completes the encoding of image acquisition results and the construction of frame sequence structure, and finally generates a set of frame images;

[0054] Edge Aggregation Submodule: Based on the frame image set, it reads image frames and converts them into grayscale channel structures. It performs positional difference calculations on the grayscale values ​​of each pixel in the continuous frame images and generates a difference layer. It sets a fixed threshold and performs binarization mapping on the grayscale difference image. It removes pixel blocks with grayscale changes below the threshold and retains the response area. It establishes a pixel adjacency index in the binary image, constructs a connected component identification map and outputs a set of boundary blocks. It parses the starting coordinates and pixel span data of each edge region in the boundary block set, extracts all boundary block position parameters and constructs a set of coordinate points. It calls the density clustering method, sets the radius parameter to 8 and the minimum number of samples to 5, performs clustering operations on the set of coordinate points and merges boundary block regions in the same cluster to generate a worm spot view.

[0055] The environmental data modeling module includes:

[0056] Parameter acquisition submodule: Based on the insect spot view, multiple sets of sensors are used to collect temperature, humidity, light and wind speed values. A multi-channel parameter stream is constructed through the time series corresponding to the sampling points. Data unit standard conversion and sampling label binding are performed and integrated into a unified structure record to obtain the original parameter set.

[0057] Feature construction submodule: Based on the original parameter set, it identifies abrupt change nodes by calculating the magnitude of numerical changes at continuous time points and performs fluctuation range location, sets upper and lower thresholds to filter out outlier records and retain stable data segments, extracts time labels for each parameter and constructs a structured sequence to obtain the environmental parameter matrix;

[0058] Parameter Acquisition Submodule: Based on the insect spot view, it uses the I2C communication protocol to read the values ​​of temperature, humidity, light, and wind speed sensors. For temperature acquisition, it calls the temperature sensor data reading command and sets the sampling interval to 10 seconds, converting the two bytes of raw data into Celsius values. For humidity acquisition, it calls the humidity sensor continuous sampling command and sets the sampling frequency to 0.5Hz, reading six bytes of data and converting the high eight bits to obtain the relative humidity percentage. For light sampling, it sets the light sensor to high-resolution mode and sets continuous sampling parameters, reading two bytes of raw data and converting it to lux units of light intensity. For wind speed acquisition, it calculates the number of rotations per unit time using the wind cup sensor pulse count, converting the pulse count into wind speed values ​​and then into meters per second. Sampling time labels are generated using a unified time format and appended to the sampling data record. All acquired data undergoes unit standardization and a unified field naming format. The sampling point data structure is stored sequentially as a dictionary structure and written into a data table, generating the raw parameter set.

[0059] Feature Construction Submodule: Based on the original parameter set, the numerical difference method is used to calculate the variation range of temperature, humidity, light intensity, and wind speed at continuous time points. A difference sequence is constructed for each parameter and the change value is recorded. The mutation identification function is called on the variation range sequence to set the minimum mutation threshold as 2 degrees for temperature, 4% for humidity, 30 lux for light intensity, and 1 meter per second for wind speed. At the mutation node, three time points are traced back and two time points are extended forward to form a fluctuation analysis interval. Within the interval, upper and lower bound screening operations are performed to filter out data records that do not conform to the variation range. The interval moving average operation is used to calculate the stability level of the parameter segment and exclude samples with frequent fluctuations. Data segments that meet the fluctuation criteria are indexed and reorganized and valid time labels are extracted. Outlier samples are identified and marked and removed using the three-standard deviation method. The remaining data is rearranged and resampled according to a one-minute time granularity and the missing fields are filled using forward padding. Finally, the time field and the four parameter fields are recombined into a standardized data structure to generate an environmental parameter matrix.

[0060] The multimodal fusion module includes:

[0061] Sequence splicing submodule: Based on the insect spot view and environmental parameter matrix, a unified time index is used to perform a one-to-one mapping between image channel values ​​and environmental parameter values. A multi-dimensional time-series vector is constructed by horizontal splicing and the source field labels are marked to obtain the time-series feature set.

[0062] Interval scaling submodule: Based on the time series feature set, it performs an interval linear transformation on the values ​​of each column of valley and peak values, adjusts the numerical amplitude through normalization mapping, and simultaneously processes missing fields and abnormal nodes to obtain a correction vector set;

[0063] Vector merging submodule: Based on the calibration vector set, a generative adversarial network is used to weight and combine each field to construct a single-channel fusion structure. The image and parameter feature vector are merged by field pairing and dimension unification processing is performed to obtain the fusion description vector.

[0064] The sequence splicing submodule, based on the worm spot view and the environmental parameter matrix, uses a unified time index to perform a one-to-one mapping between image channel values ​​and environmental parameter values. It reads the timestamp field of each row in the image channel matrix and performs equality matching with the timestamp field in the environmental parameter matrix, filters out record pairs with completely identical timestamps, sorts the matching results in chronological order and reconstructs the index order, splices the environmental parameter fields according to the image channel arrangement order, uses horizontal connection for splicing and uses the field source identifier as the column name prefix, numbers each field of the merged field set and marks the source label, and generates a time series feature set.

[0065] The interval scaling submodule, based on the time series feature set, performs normalization mapping on all fields using interval linear transformation. It iterates through all field values ​​column by column and records the minimum and maximum values ​​in each column as boundaries. It subtracts the minimum value from each field value and divides it by the difference between the maximum and minimum values. It retains three decimal places for all calculation results and replaces the original field values. It performs linear interpolation on missing fields and sets the interpolation window size to the two time points before and after. For fields with outlier node values, it uses a truncation method to limit the defined maximum and minimum value intervals. It converts the processed field matrix into a standardized structure and generates a correction vector set.

[0066] The vector merging submodule, based on the calibration vector set, employs a generative adversarial network (GAN) for vector fusion processing. First, a fusion structure generator model is constructed, with the input dimension set to the calibration vector dimension. Following a three-layer structure, an input layer, a feature extraction layer, and an output layer are constructed sequentially, with 64, 32, and 1 nodes per layer, respectively. The activation function is set to a linear function, and batch normalization is enabled. Weighted parameters for each input field are set to a range of 0 to 1, and a weighted summation operation is performed to generate a single-channel fusion feature structure. A discriminator model is constructed, with a two-layer structure, and the fusion structure vector and the true label are input. The discriminant output is set to a probability value, and the generator parameters are updated based on the error calculation. The generation and discriminant steps are iteratively executed to complete the training and convergence of the fusion structure. The fusion output result is structurally adjusted and its dimensional format unified to generate a fusion descriptive vector.

[0067] Generative Adversarial Networks (GANs) first initialize the parameters of the generator and discriminator networks. The calibration vector set is then input into the generator, which sequentially performs multi-layer linear mapping and activation transformation, outputting a fused candidate vector. The fused candidate vector and the real temporal feature vector samples are then input into the discriminator. The discriminator calculates the discrimination scores for real and fake samples through a classification layer. Next, the adversarial loss is calculated based on the discriminator output and the real labels. Gradient backpropagation and parameter updates are then performed on both the discriminator and the generator. Finally, the above generation and discrimination loop is repeated until the misclassification rate of the discriminator for the generated samples reaches a preset standard.

[0068] Generate adversarial networks according to the formula:

[0069]

[0070] Where: v′ f Let W represent the fusion description vector, σ represent the activation function, τ represent the temperature coefficient, and W represent the temperature coefficient. c Let v represent the fusion transformation weight matrix, α represent the fusion weights of image and parameter features, and v img The feature vector representing the crop pest image field, v param This represents the feature vector of environmental parameters associated with pests, γ represents the weighting coefficient of the auxiliary feature vector, and v aux This represents the contextual auxiliary features generated by the multimodal large language model, where η represents the gating coefficient between the image and the parameter field, and b c δ represents the fusion bias term, and δ represents the bias correction term;

[0071] Execution process: First, the visual encoder extracts features from the pest image, generating an image field feature vector v. img Simultaneously, environmental parameter data associated with pest occurrences, including temperature, humidity, soil pH, and rainfall, are collected and input into a parameter encoding network to generate a structured parameter feature vector v. param Next, the fusion weight α between the image and parameters is learned through the adversarial discriminant module, completing the weighted combination of image and parameter fields. Then, a multimodal large language model is used to encode unstructured text information such as crop type, growth stage, and historical pest records to generate context auxiliary vector v. aux Subsequently, element-wise multiplication of the image with the parameter feature vector is performed to represent the fine-grained coupling relationship between the visual features of pests and environmental variables. This is then multiplied by a gating coefficient η to regulate the intensity of the influence. The above four types of features are weighted proportionally and fed into a fusion mapping network for linear transformation. The weight matrix is ​​W. c , fusion bias term b cIn addition, a bias correction term δ is added to correct the potential input skewness distribution. The entire linear output result is scaled by a temperature factor τ to optimize the gradient distribution and numerical stability. Finally, it is input into the activation function σ to generate the fused descriptor vector v′. f This serves as the input for the subsequent pest classification and early warning generation module.

[0072] The intelligent classification module includes:

[0073] Distance calculation submodule: Based on the fused description vector, it performs item-by-item difference operation with the center vector of each category by using the coordinate correspondence between vectors, and obtains the Euclidean distance by accumulating the squared differences and outputs the distance set to obtain the distance distribution;

[0074] The tag extraction submodule retrieves the corresponding tag number based on the distance distribution using the minimum distance value index, obtains the category name by matching the tag mapping table, and combines them to form a standard classification identifier, thus obtaining the initial tag.

[0075] The central update submodule is based on the fused description vector and initial label. It adopts an isolated forest. When the distance valley value is higher than the set threshold, it extracts the difference between the vector and the original central vector and performs coordinate accumulation. At the same time, it adds the corresponding number to the label sequence to obtain the pest category label.

[0076] The distance calculation submodule, based on the fused description vector, performs item-by-item difference operations with each category center vector using the coordinate correspondence between vectors. It performs numerical subtraction operations on the same dimension for each group of vectors and records the difference results. It iterates through all difference elements and performs a squaring operation. It sums all the squared values ​​and uses the square root method to calculate the square root of the sum of squares as the distance value of a single category center. It iterates through all category center calculation operations and writes each group of distance values ​​into the distance sequence set in order, finally generating the distance distribution.

[0077] Tag extraction submodule: Based on distance distribution, it adopts the minimum distance value index retrieval method, selects the minimum distance value from the distance sequence set and extracts the corresponding index position, uses the index value as the tag number to perform a search and matching operation in the preset tag dictionary, obtains the text identifier field of the corresponding tag, concatenates the tag number and the tag identifier field to construct a standardized tag structure, and finally generates the initial tag;

[0078] The central update submodule, based on the fused descriptive vector and initial labels, performs anomaly detection using the isolated forest method. It constructs a sample input set by combining the current fused descriptive vector with the original category center vector. By building multiple sub-sampling binary trees, it sequentially divides the sample space and records the path length. It calculates the average path length of the current input sample and compares it with a judgment threshold. When the path length is less than the threshold, it is identified as an anomaly vector. It performs a dimension-by-dimensional difference operation on the anomaly vector and the center vector, and accumulates the result with the original center vector to form a new center vector. At the same time, it assigns a new category number to the current fused descriptive vector and adds it to the label number sequence set, finally generating pest category labels.

[0079] Isolation Forest first constructs multiple random subsets of samples and trains binary split trees independently on each subset. During the tree construction process, feature fields and split thresholds are randomly selected to divide the samples. Then, the input fused description vector is traversed through all trees, and the path length to reach the leaf node in each tree is recorded. The average path length in all trees is then used as the evaluation criterion. Based on the set anomaly threshold, it is determined whether the vector deviates from the current classification center. If the deviation condition is met, the difference between the vector and the center vector is extracted, the center position is updated, and a new label number is appended to generate the pest category label.

[0080] An isolated forest, according to the formula:

[0081]

[0082] Among them: I i The improved distance index represents the distance between the fused description vector and the center of the i-th type of pest, where v represents the pest description vector obtained by fusing image, parameters, and linguistic semantics, and c represents the distance between the fused description vector and the center of the i-th type of pest. i This represents the current center vector of the i-th type of pest. Let α represent the inverse of the covariance matrix of the i-th type of pest sample. ′ β represents the coefficient of the structural correlation weighting term, P represents the spatial structural correlation matrix of crop pests, and β represents the coefficient of the structural correlation weighting term. ′ γ represents the constraint coefficient of the auxiliary vector on the fusion result, z represents the auxiliary semantic vector output by the anomaly detection tree in the isolated forest model, and γ represents the constraint coefficient of the auxiliary vector on the fusion result. ′ θ represents the regularization constant, and θ represents the preset label update threshold bias term;

[0083] Execution process: First, image fields, environmental parameter fields, and crop semantic information are received. A pest fusion description vector v is generated through a fusion model. Then, the current i-th class center vector c is extracted from the established pest class center set. i Calculate v and c i The difference between them, combined with the inverse of the sample covariance matrix After calculating the Mahalanobis distance term, a structured quadratic form operation is performed on the above difference using the structure correlation matrix P, and then multiplied by the structure weighting coefficient α′ to reflect the typical spatial pattern differences of pest categories. Next, an auxiliary semantic vector z is obtained from multiple random anomaly detection trees in an isolated forest, and the squared Euclidean distance between z and the fused descriptive vector v is calculated. A weighting coefficient β′ is introduced to control the influence range of the auxiliary term. Subsequently, the above three terms are summed, and a regularization constant γ′ and a threshold bias term θ are added. The weighted sum is then squared to obtain the final distance index I. i If the difference is greater than the preset threshold, the coordinate difference between the current vector and the center vector is accumulated and updated, and the corresponding number of the new sample is appended to the label sequence, so as to dynamically obtain and maintain the pest category label set.

[0084] The alarm scheduling module includes:

[0085] Template Filling Submodule: Based on pest category tags, it uses a mapping table to retrieve category numbers and extract corresponding message templates. It replaces the tag numbers and category names in the fields and injects the template text. It then concatenates message paragraphs and delimiters to build a push list and obtain the push content set.

[0086] Push execution submodule: Based on the push content set, it calls the SMS interface to submit each text message and receive the status response by traversing the list, and calls the application interface to receive the confirmation information by traversing the same text message. It also writes the status response and confirmation information to the log and updates the task status, and obtains the alarm task list.

[0087] Template Filling Submodule: Based on pest category tags, it uses key-value mapping to retrieve the message template corresponding to the tag number. It uses field replacement instructions to find all the markers enclosed in double curly braces in the template text and records the list of marker names. It reads the tag number and category name fields one by one and performs string replacement operations on the markers. It appends the replaced text to the paragraph content buffer, calls the connection instruction to merge the content of each field in paragraph order and inserts "--" as a separator between paragraphs, builds a list of multiple merged message structures, records each message structure as a separate text entry and adds it to the push cache container, and finally generates the push content set.

[0088] The push execution submodule, based on the push content set, uses a sequential traversal mechanism to call the external SMS interface. Each push text is submitted to the specified interface address via a POST request, with headers parameters including content-type as application / json and authorization as a token value. The message content field is passed through the body structure, and the interface's return status code is received. The submission response time and return status are recorded and written to a log file. The application interface is called to execute the same text submission process, setting the interface address to the application push path and configuring the POST method and request header content. The parameters maintain the same format as the SMS interface, and the returned confirmation information is written to the log file. The status of each response in the log file is read, and the task execution status field is set to completed or failed. All task statuses are written to the task tracking table, ultimately generating an alarm task list.

[0089] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A smart identification and early warning system for crop pests based on a multimodal large language model, characterized in that, The system includes: The lesion capture module acquires a panoramic view of the plant through a camera, collects continuous frames of leaf surface images, calculates color differences and performs edge filtering, performs connected component judgment and region clustering screening, and obtains insect spot views. Environmental data modeling module: Based on the insect spot view, collect temperature, humidity, light and wind speed data, filter out abnormal sections through difference calculation and fluctuation detection, extract time-related feature values, and form an environmental parameter matrix; Multimodal fusion module: Based on the insect spot view and environmental parameter matrix, the image channel values ​​and environmental parameter values ​​are concatenated in chronological order. A generative adversarial network is used to perform numerical range scaling and dimension correction, complete scale unification and vector combination, and obtain the fusion description vector. Intelligent classification module: Calculates the Euclidean distance from the fused description vector to the center vector of each category, selects matching labels according to the center index corresponding to the distance, and adopts isolated forest. If the distance exceeds the threshold, it extracts the difference vector between the current vector and the neighboring center, superimposes it on the original center coordinates and appends the label number to generate pest category labels. Alarm scheduling module: Based on the pest category tags, it matches the level number with the notification template, fills the push text with formatted fields, calls the SMS and application interfaces and sends the message, collects the response and writes it to the log, and generates an alarm task list.

2. The intelligent identification and early warning system for crop pests based on a multimodal large language model according to claim 1, characterized in that, The lesion capture module includes: Image acquisition submodule: Uses multi-angle cameras to acquire panoramic images of the plant and capture continuous frame images of the leaf surface. By completing image acquisition within a fixed time interval and constructing a frame sequence structure, and performing pixel matrix reading and timestamp recording, the frame image number and sampling batch identifier are completed to obtain a set of frame images; Edge aggregation submodule: Based on the frame image set, it calculates the adjacent difference layer by extracting the gray value of each pixel in the image, filters out low-change areas and retains the edge response area, establishes a connected graph through the adjacent pixel index and generates a set of boundary blocks, and performs spatial clustering to merge similar areas in combination with the position coordinates to obtain the worm spot view.

3. The intelligent identification and early warning system for crop pests based on a multimodal large language model according to claim 1, characterized in that, The environmental data modeling module includes: Parameter acquisition submodule: Based on the insect spot view, multiple sets of sensors are used to collect temperature, humidity, light and wind speed values. A multi-channel parameter stream is constructed through the time series corresponding to the sampling points. Data unit standard conversion and sampling label binding are performed and integrated into a unified structure record to obtain the original parameter set. Feature construction submodule: Based on the original parameter set, identify mutation nodes by calculating the magnitude of numerical changes at continuous time points and perform fluctuation range positioning, set upper and lower thresholds to filter out outlier records and retain stable data segments, extract time labels for each parameter and construct structured sequences to obtain the environmental parameter matrix.

4. The intelligent identification and early warning system for crop pests based on a multimodal large language model according to claim 1, characterized in that, The multimodal fusion module includes: Sequence splicing submodule: Based on the insect spot view and environmental parameter matrix, a unified time index is used to perform a one-to-one mapping between image channel values ​​and environmental parameter values. A multi-dimensional time-series vector is constructed by horizontal splicing and the source field labels are marked to obtain the time-series feature set. Interval scaling submodule: Based on the time series feature set, it performs an interval linear transformation on the logarithmic values ​​of each column of valley and peak values, adjusts the numerical amplitude through normalization mapping, and simultaneously processes missing fields and abnormal nodes to obtain a correction vector set; Vector merging submodule: Based on the correction vector set, a generative adversarial network is used to weight and combine each field to construct a single-channel fusion structure. The image and parameter feature vector are merged by field pairing and dimensionality unification processing is performed to obtain the fusion description vector.

5. The intelligent identification and early warning system for crop pests based on a multimodal large language model according to claim 4, characterized in that, The generative adversarial network (GAN) first initializes the parameters of the generator and discriminator networks. The calibration vector set is then input into the generator, which sequentially performs multi-layer linear mapping and activation transformation, outputting a fused candidate vector. The fused candidate vector and the real temporal feature vector samples are then input into the discriminator. The discriminator calculates the discrimination scores for real and fake samples through a classification layer. Next, the adversarial loss is calculated based on the discriminator output and the real labels. Gradient backpropagation and parameter updates are then performed on both the discriminator and the generator. Finally, the above generation and discrimination loop is repeated until the misclassification rate of the discriminator for the generated samples reaches a preset standard.

6. The intelligent identification and early warning system for crop pests based on a multimodal large language model according to claim 4, characterized in that, The generative adversarial network is defined according to the formula: Where: v′ f Let W represent the fusion description vector, σ represent the activation function, τ represent the temperature coefficient, and W represent the temperature coefficient. c Let v represent the fusion transformation weight matrix, α represent the fusion weights of image and parameter features, and v img The feature vector representing the crop pest image field, v param This represents the feature vector of environmental parameters associated with pests, γ represents the weighting coefficient of the auxiliary feature vector, and v aux This represents the contextual auxiliary features generated by the multimodal large language model, where η represents the gating coefficient between the image and the parameter field, and b c δ represents the fusion bias term, and δ represents the bias correction term.

7. The intelligent identification and early warning system for crop pests based on a multimodal large language model according to claim 1, characterized in that, The intelligent classification module includes: Distance calculation submodule: Based on the fused description vector, it performs item-by-item difference operation with the center vector of each category using the coordinate correspondence between vectors, calculates the Euclidean distance by accumulating the squared differences, and outputs the distance set to obtain the distance distribution; Tag extraction submodule: Based on the distance distribution, the corresponding tag number is retrieved using the minimum distance value index, the category name is obtained by matching the tag mapping table and combined to form a standard classification identifier, and the initial tag is obtained; The central update submodule: Based on the fused description vector and the initial label, it adopts an isolated forest. When the distance valley value is higher than the set threshold, it extracts the difference between the vector and the original central vector and performs coordinate accumulation. At the same time, it adds the corresponding number to the label sequence to obtain the pest category label.

8. The intelligent identification and early warning system for crop pests based on a multimodal large language model according to claim 7, characterized in that, The isolated forest is constructed by first building multiple random subsample sets and independently training binary split trees on each subsample set. During the tree construction process, feature fields and split thresholds are randomly selected to divide the samples. Then, the input fused description vector is traversed through all trees, and the path length to reach the leaf node in each tree is recorded. The average path length in all trees is then used as the evaluation criterion. Based on the set anomaly threshold, it is determined whether the vector deviates from the current classification center. If the deviation condition is met, the difference between the vector and the center vector is extracted, the center position is updated, and a new label number is added to generate the pest category label.

9. The intelligent identification and early warning system for crop pests based on a multimodal large language model according to claim 7, characterized in that, The isolated forest, according to the formula: Among them: I i The improved distance index represents the distance between the fused description vector and the center of the i-th type of pest, where v represents the pest description vector obtained by fusing image, parameters, and linguistic semantics, and c represents the distance between the fused description vector and the center of the i-th type of pest. i This represents the current center vector of the i-th type of pest. Let α′ represent the inverse of the covariance matrix of the i-th type of pest sample, α′ represent the coefficient of the structural correlation weighting term, P represent the spatial structural correlation matrix of crop pests, β′ represent the constraint coefficient of the auxiliary vector on the fusion result, z represent the auxiliary semantic vector output by the anomaly detection tree in the isolated forest model, γ′ represent the regularization constant, and θ represent the preset label update threshold bias term.

10. The intelligent identification and early warning system for crop pests based on a multimodal large language model according to claim 1, characterized in that, The alarm scheduling module includes: Template filling submodule: Based on the pest category tags, the module retrieves the category number using a mapping table and extracts the corresponding message template. It then replaces the tag number and category name in the field and injects the template text. Finally, it concatenates message paragraphs and separators to construct a push list and obtain the push content set. Push execution submodule: Based on the push content set, it calls the SMS interface to submit each text message and receive the status response by traversing the list, and calls the application interface to receive the confirmation information by traversing the same text message. It also writes the status response and confirmation information to the log and updates the task status, and obtains the alarm task list.

Images