Crop intelligent identification system based on CLIP multi-mode method

Abstract

The invention discloses a system for realizing crop intelligent identification based on a CLIP multi-mode method, and belongs to the technical field of crop identification, and the method specifically comprises the steps: collecting real-time images, text description and environment data of crops; performing image processing on the real-time image; a CLIP model is used for carrying out feature extraction on the image and text description, visual features are extracted from the image, semantic features are extracted from the text, the text description comprises crop types and symptom description of diseases and pests, the environment data and the visual features are fused, and a multi-modal feature vector is generated; the multi-modal feature vectors are classified by using a pre-trained classifier, specific crop types and pest and disease damage types are identified according to extracted multi-modal features, identification results are displayed to users through a visual interface, and information is fed back to a monitoring center; according to the method, visual, semantic and environmental features are fused, so that the accuracy of crop disease and insect pest recognition is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of crop identification, and in particular to a CLIP-based multimodal method for realizing an intelligent crop identification system. Background Art

[0002] With the development of modern agriculture, high yield and quality of crops have become important goals of agricultural production. However, the occurrence of pests and diseases poses a serious threat to crop yield and quality. Traditional pest and disease monitoring mainly relies on manual inspections, which is inefficient and easily affected by subjective factors.

[0003] In recent years, with the development of computer vision and artificial intelligence technology, it has become a trend to use high-point surveillance video images combined with image algorithms to identify crop pests and diseases. However, the accuracy of existing methods still needs to be improved when processing images in complex environments, and the fusion of multimodal data is insufficient, which limits the practicality and reliability of the system. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal method based on CLIP to realize an intelligent crop recognition system and solve the following technical problems: The accuracy of existing methods when processing images in complex environments still needs to be improved, and the fusion of multimodal data is insufficient, which limits the practicality and reliability of the system.

[0005] The purpose of the present invention can be achieved through the following technical solutions: A multimodal method based on CLIP is used to realize an intelligent crop recognition system, including: A data collection module, used to collect real-time images, text descriptions and environmental data of crops, wherein the text descriptions include descriptions of the types of crops and symptoms of pests and diseases; An image processing module is used to perform noise reduction on the real-time image, adjust the contrast and brightness of the real-time image to a set value, and distinguish the crop area from the background area in the real-time image based on image segmentation technology; The feature extraction module is used to extract features from real-time images and text descriptions based on the CLIP model, extract visual features from real-time images, extract semantic features from text descriptions, fuse environmental data with visual features, and generate multimodal feature vectors; The feature classification module is used to classify the multimodal feature vectors using a pre-trained classifier and identify the types of crops and the corresponding pests and diseases based on the extracted multimodal features; The result output module is used to display the recognition results to the user through a visual interface and feed back the information to the monitoring center.

[0006] As a further solution of the present invention: in the data acquisition module, the environmental data includes but is not limited to temperature, humidity, and soil nutrient concentration.

[0007] As a further solution of the present invention: in the image processing module, the process of noise reduction processing is: The real-time image is converted into a grayscale image, and the size of the grayscale image is adjusted to a set specification, and then the grayscale is wavelet transformed to decompose the grayscale image into several scales. The grayscale image is decomposed into several wavelet coefficients, and the wavelet coefficients represent the local features of the image at different scales. The wavelet coefficients of the high-frequency sub-band are threshold processed, and the noise coefficient is suppressed by the set threshold, and the signal coefficient is retained. After the threshold processing, the image is reconstructed based on the retained signal coefficient and wavelet coefficient, and the processed coefficient is converted back to the spatial domain through the inverse wavelet transform to obtain the denoised grayscale image and the real-time image.

[0008] As a further solution of the present invention: In the feature extraction module, the specific process of extracting features by the CLIP model is as follows: The CLIP model includes an image encoder and a text encoder. The image encoder is based on a ResNet network and converts an input real-time image into a visual feature vector and extracts visual features from the visual feature vector. The text encoder is based on a Transformer structure and is used to convert an input text description into a text feature vector and capture semantic features in the text from the text feature vector. The pre-paired visual feature vectors and text feature vectors are marked as positive sample pairs, and the unpaired ones are marked as negative sample pairs; the CLIP model embeds the visual feature vectors and text feature vectors into a shared semantic space, calculates the similarity of the visual feature vectors and the text feature vectors in the semantic space, and adjusts the model parameters according to the calculation results so that the similarity of the positive sample pairs approaches 1 and the similarity of the negative sample pairs approaches 0. The CLIP model uses a symmetric loss function, which consists of two parts of cross entropy. The corresponding loss is calculated from the row direction and the column direction respectively, and finally the average of the loss in the two directions is taken.

[0009] As a further solution of the present invention: for each real-time image, the model calculates the similarity between its visual feature vector and all text feature vectors, and converts all similarities into probability distributions through the softmax function; for each text description, the model calculates the similarity between its text feature vector and all visual feature vectors, and minimizes the inner product with irrelevant features. During the training process, the parameters are continuously adjusted to minimize the value of the loss function until convergence.

[0010] As a further solution of the present invention: the data acquisition module is also used to perform unit conversion and normalization processing on all environmental data, and the normalization formula is x' = (x - min(x)) / (max(x) - min(x)), where x' represents the normalized data and x represents the original data.

[0011] As a further solution of the present invention: the data acquisition module is also used to perform deduplication processing on the text description, for any pair of text descriptions with different dates but the same number of words, they are marked as the first text and the second text respectively, the occurrence frequency of preset keywords of the text descriptions are collected respectively, the number of the top three keywords with the highest occurrence frequency is collected, the number of occurrences of the top three keywords of the first text is marked as n1, n2 and n3, the number of occurrences of the top three keywords of the second text is marked as m1, m2 and m3, the three values ​​are used as the upper base, lower base and height of the trapezoid respectively, the first trapezoid and the second trapezoid are generated respectively, the area S1=(n1+n2)n3 / 2 of the first trapezoid is calculated, the area S2=(m1+m2)m3 / 2 of the first trapezoid is calculated, the first trapezoid and the second trapezoid are superimposed on a plane, the superposition state when the superposition area is the largest is selected, the area S3 of the superposition area under the current superposition state is obtained, the value of S3 / (S1+S2) is calculated, which is the shape similarity, two text descriptions with a similarity greater than 90% are marked as duplicate texts, and the text description with the later date is deleted.

[0012] As a further solution of the present invention: the process of fusing features in the feature extraction module is as follows: An encoder is used to extract features from environmental data, and the numerical features of environmental data are extracted based on the attention mechanism. The features are mapped into vector form in the multimodal space, and the extracted visual features and numerical features are weightedly fused. Different weights are assigned to the visual features and numerical features, and then the weighted sum is used to obtain the comprehensive features.

[0013] As a further solution of the present invention: the pre-training process of the classifier is: Mark and annotate the image data, text data, and environmental data, clean, denoise, and standardize the collected data, use convolutional neural networks as classifiers, and select models trained on large-scale datasets as the base model. Freeze the convolutional layers of the base model and only train the newly added classifiers. After the classifier training is completed, unfreeze the convolutional layers of the base model and continue to train the entire model based on the collected data. Use the annotated dataset to train the model and evaluate it using evaluation indicators, including accuracy, recall, and F1 scores. During the training process, continuously adjust the parameters to minimize the value of the loss function until the model converges.

[0014] Beneficial effects of the present invention: (1) The present invention significantly improves the accuracy of crop pest and disease identification by integrating visual, semantic and environmental features. Compared with the prior art, the present invention introduces the CLIP model, which can simultaneously process information described by images and natural language. The CLIP model solves the limitation of traditional methods that rely only on single-modal data in complex environments by comparative learning of visual features and text semantics. For example, traditional pest and disease identification technology usually relies only on image data, which has poor recognition effect in complex scenes such as lighting changes, weather influences or crop occlusion. By combining image visual features with text semantic features, the present invention can rely on semantic level supplementation when image information is insufficient, thereby enhancing the model's understanding ability. At the same time, the introduction of environmental data allows the system to consider multiple factors when analyzing the occurrence of crop pests and diseases, thereby making more accurate judgments. This innovation of multimodal data fusion significantly improves the accuracy and robustness of recognition, allowing the system to operate stably in a variety of complex environments.

[0015] (2) The present invention integrates multimodal features to enable the system to maintain high recognition stability in the face of different environmental conditions, such as light intensity, weather changes, and crop occlusion. Current crop pest and disease recognition systems perform poorly in complex environments, especially when changes in light and weather conditions can easily lead to a significant decrease in recognition accuracy. In the prior art, many systems often make misjudgments or miss judgments due to the decline in image quality when faced with these changes. The present invention introduces the CLIP model's multimodal feature extraction and contrast learning, which not only relies on image quality, but also effectively overcomes the influence of these unfavorable factors by integrating environmental data and crop text descriptions. The system can maintain high recognition accuracy under different light, weather, and crop growth cycles, especially under low light or bad weather conditions where traditional methods are prone to errors. The system of the present invention can still maintain reliable performance. Therefore, the present invention has significantly improved the system's robustness and adaptability, and can adapt to various complex farmland environments.

[0016] (3) The present invention can achieve real-time monitoring and rapid response of large-scale crops by combining high-point monitoring equipment with efficient algorithms. Compared with traditional crop monitoring methods that rely on manual inspections or single image recognition systems, the system of the present invention can continuously obtain high-definition images of large-scale farmland and conduct real-time analysis and feedback in combination with environmental data. In the prior art, the response speed of real-time monitoring systems is often limited by computing resources and algorithm efficiency, and it is impossible to conduct continuous and accurate pest and disease monitoring in large-scale crop areas. The present invention optimizes the data acquisition and processing process, uses the CLIP model to efficiently extract multimodal features, and uses pre-trained classifiers to quickly classify pest and disease types, thereby greatly improving the real-time performance of the system. In addition, the system can also provide timely feedback to the monitoring center based on the recognition results, helping agricultural managers to take quick measures to reduce the damage of pests and diseases to crops. This real-time monitoring and response capability not only improves the efficiency of crop management, but also reduces crop losses caused by delays. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention will be further described below in conjunction with the accompanying drawings.

[0018] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] See also Figure 1 As shown, the present invention is a multimodal method based on CLIP to realize an intelligent crop recognition system, which specifically includes: 1. Data Collection The present invention first obtains high-definition images of crops in real time through high-point monitoring equipment, and simultaneously collects relevant environmental data. Environmental data include but are not limited to temperature, humidity, soil nutrient concentration, etc., which provide important contextual information for subsequent identification of pests and diseases. Through the deployment of high-point monitoring equipment, a large area of ​​farmland can be covered to monitor the growth of crops and the occurrence of pests and diseases in real time. The acquisition of high-definition images provides high-quality raw data for subsequent image processing and feature extraction, while environmental data provides a more comprehensive reference for identification of pests and diseases.

[0021] 2. Image Preprocessing Due to the complex farmland environment, the collected images may be affected by factors such as weather, lighting, and noise, so the preprocessing step is very necessary. The image preprocessing module performs denoising, enhancement, and segmentation on the acquired images. First, denoising can effectively reduce random noise in the image, such as factors such as uneven lighting and shadows; second, enhancement processing improves the contrast and brightness of the image, making the characteristics of pests and diseases more prominent; finally, image segmentation technology can distinguish the crop area from the background area in the image, further improving the accuracy of feature extraction. This process greatly improves the effectiveness and accuracy of subsequent feature extraction.

[0022] 3. Multimodal feature extraction After image preprocessing, the system enters the multimodal feature extraction stage. The core innovation of this stage is to use the CLIP model to extract features from images and text descriptions. The CLIP model is a pre-trained multimodal neural network that can process image and text information simultaneously. Specifically, the CLIP model extracts visual features from images and semantic features from text. The text description can include semantic information such as the type of crop and the symptom description of pests and diseases. These text information and the visual features in the image form the basis for contrastive learning.

[0023] In addition to visual features and semantic features, the present invention also fuses environmental data (such as temperature, humidity, soil nutrient concentration, etc.) with visual features to form a comprehensive multimodal feature vector. This fusion of multimodal features greatly improves the accuracy of pest and disease identification. Environmental data is closely related to the occurrence of crop pests and diseases. With the help of these data, the system can better understand the conditions for the occurrence of pests and diseases and make more accurate identification and diagnosis.

[0024] 4. Feature matching and classification After the multimodal feature extraction is completed, the system uses a pre-trained classifier to classify the multimodal feature vector. The classifier in the present invention can be a support vector machine (SVM), a random forest or a deep neural network algorithm. The role of the classifier is to identify specific crop types and their pests and diseases based on the extracted multimodal features.

[0025] The present invention also introduces a contrastive learning method to further optimize feature representation. The core idea of ​​contrastive learning is to optimize the feature extraction capability of the model by comparing different samples. In the technical solution of the present invention, contrastive learning is trained on different images and descriptions of the same crop, so that the model can better capture the correlation between the image and the text, thereby improving the recognition accuracy. Feature optimization is particularly important in complex environments, such as lighting changes, crop occlusion, etc.

[0026] 5. Result output and feedback After the classification is completed, the system will display the recognition results to the user through a visual interface. The visual interface can intuitively display the types of crops, the types of pests and diseases, and the corresponding environmental data. Users can understand the real-time situation of the farmland through the interface and make corresponding management decisions. In addition, the system will also feed back the recognition information to the monitoring center to realize intelligent crop management. The monitoring center can take corresponding measures in time according to the recognition results, such as spraying pesticides, adjusting water, etc., so as to reduce the harm of pests and diseases to crops.

[0027] The core of this invention is to use the CLIP model for multimodal feature extraction, and to form a comprehensive multimodal feature vector by comparing visual features and text semantic features, combined with additional information such as temperature, humidity, soil nutrient concentration, etc. in the farmland environment. This fusion method greatly improves the system's recognition ability in complex and changing environments, especially in the case of uneven lighting, many obstructions, or poor weather conditions, and can still accurately identify the types of crop pests and diseases.

[0028] CLIP model application: CLIP learns the correspondence between vision and language by jointly training images and text. When applied to crop pest and disease identification, the system can compare crop images with corresponding semantic descriptions to improve the ability to understand objects in the image. Compared with traditional image classification methods, CLIP can not only identify visual features in images, but also combine semantic information for a deeper understanding, allowing the system to perform well in a wider range of scenarios.

[0029] Introduction of environmental data: The present invention specifically considers the introduction of agricultural environmental data, such as temperature, humidity, soil data, etc. These data, combined with visual features, provide additional contextual information for the identification of crop pests and diseases. For example, the occurrence of certain pests and diseases is closely related to specific climatic conditions. The introduction of environmental data can help the system better predict and identify these pests and diseases, further improving the practicality and accuracy of the system.

[0030] Contrastive learning optimization: In order to further improve the recognition accuracy, the present invention also adopts the contrastive learning method, which optimizes the representation effect of the feature vector by comparing the feature representation under different modalities. This method can reduce error accumulation, especially when processing multimodal data fusion, and further improve the accuracy and robustness of the classifier.

[0031] In another preferred embodiment of the present invention, in the data acquisition module, the environmental data includes but is not limited to temperature, humidity, and soil nutrient concentration.

[0032] In another preferred embodiment of the present invention, in the image processing module, the noise reduction process is: 1. Convert to grayscale image: First, the real-time acquired color image is converted into a grayscale image. This conversion removes color information and retains only brightness information, thereby simplifying subsequent processing steps and reducing the amount of calculation.

[0033] Grayscale processing can be achieved through a variety of methods, such as taking the average or weighted average of the three RGB channels.

[0034] 2. Adjust the image size: According to actual needs, adjust the size of the grayscale image to the preset specifications. This step is usually to ensure that the image meets the input requirements of a specific algorithm or model, or to optimize storage and transmission efficiency.

[0035] Resizing can be achieved through techniques such as interpolation, cropping, or scaling.

[0036] 3. Wavelet transform: The adjusted grayscale image is subjected to wavelet transform, which is a multi-scale analysis tool that can decompose the image into components of different frequencies.

[0037] The result of wavelet transform is to represent the original image as a series of wavelet coefficients, which represent the local features of the image at different scales.

[0038] 4.Threshold processing: Among the wavelet coefficients obtained by wavelet transform, the wavelet coefficients of high-frequency subbands are often related to image details and noise. Therefore, threshold processing of these high-frequency coefficients is a key step in denoising.

[0039] A suitable threshold is set to suppress the noise coefficients below the threshold while retaining the signal coefficients above the threshold. This can effectively remove noise while retaining the important features of the image.

[0040] 5. Image reconstruction: Image reconstruction is performed based on the signal coefficients and wavelet coefficients retained after threshold processing. This step is achieved through inverse wavelet transform, which converts the processed coefficients back to the spatial domain to form a denoised grayscale image.

[0041] Compared with the original image, the noise of the reconstructed grayscale image is significantly suppressed while retaining more useful information.

[0042] 6. Output results: The final denoised grayscale image is converted into a denoised real-time image, which can be used for subsequent analysis, recognition or other processing tasks.

[0043] In another preferred embodiment of the present invention, in the feature extraction module, the specific process of extracting features by the CLIP model is: The CLIP model includes an image encoder and a text encoder. The image encoder is based on a ResNet network and converts an input real-time image into a visual feature vector and extracts visual features from the visual feature vector. The text encoder is based on a Transformer structure and is used to convert an input text description into a text feature vector and capture semantic features in the text from the text feature vector. The pre-paired visual feature vectors and text feature vectors are marked as positive sample pairs, and the unpaired ones are marked as negative sample pairs; the CLIP model embeds the visual feature vectors and text feature vectors into a shared semantic space, calculates the similarity of the visual feature vectors and the text feature vectors in the semantic space, and adjusts the model parameters according to the calculation results so that the similarity of the positive sample pairs approaches 1 and the similarity of the negative sample pairs approaches 0. The CLIP model uses a symmetric loss function, which consists of two parts of cross entropy. The corresponding loss is calculated from the row direction and the column direction respectively, and finally the average of the loss in the two directions is taken.

[0044] In another preferred embodiment of the present invention, for each real-time image, the model calculates the similarity between its visual feature vector and all text feature vectors, and converts all similarities into probability distributions through a softmax function; for each text description, the model calculates the similarity between its text feature vector and all visual feature vectors, and minimizes the inner product with irrelevant features. During the training process, the parameters are continuously adjusted to minimize the value of the loss function until convergence.

[0045] In another preferred embodiment of the present invention, the data acquisition module is also used to perform unit conversion and normalization processing on all environmental data. The normalization formula is x' = (x - min(x)) / (max(x) - min(x)), where x' represents the normalized data and x represents the original data.

[0046] In another preferred embodiment of the present invention, the data acquisition module is also used to perform deduplication processing on the text description, for any pair of text descriptions with different dates but the same number of words, they are marked as the first text and the second text respectively, the occurrence frequency of preset keywords of the text descriptions is collected respectively, the number of the top three keywords with the highest occurrence frequency is collected, the number of occurrences of the top three keywords of the first text is marked as n1, n2 and n3, the number of occurrences of the top three keywords of the second text is marked as m1, m2 and m3, the three values ​​are used as the upper base, lower base and height of the trapezoid respectively, the first trapezoid and the second trapezoid are generated respectively, the area S1=(n1+n2)n3 / 2 of the first trapezoid is calculated, the area S2=(m1+m2)m3 / 2 of the first trapezoid is calculated, the first trapezoid and the second trapezoid are superimposed on the plane, the superposition state when the superposition area is the largest is selected, the area S3 of the superposition area under the current superposition state is obtained, the value of S3 / (S1+S2) is calculated, which is the shape similarity, two text descriptions with a similarity greater than 90% are marked as duplicate texts, and the text description with the later date is deleted.

[0047] In another preferred embodiment of the present invention, the feature extraction module, the process of fusing features is: An encoder is used to extract features from environmental data, and the numerical features of environmental data are extracted based on the attention mechanism. The features are mapped into vector form in the multimodal space, and the extracted visual features and numerical features are weightedly fused. Different weights are assigned to the visual features and numerical features, and then the weighted sum is used to obtain the comprehensive features.

[0048] In another preferred embodiment of the present invention, the pre-training process of the classifier is: 1. Data preparation and preprocessing: First, collect a large amount of labeled image data, text data, and environmental data. These data are the basis for model training and need to cover the conditions of various crop diseases and pests, as well as related environmental factors such as temperature, humidity, soil nutrients, etc.

[0049] The collected data is cleaned, denoised and standardized. This step is to eliminate noise and outliers in the data, improve data quality, and convert the data into a format suitable for model processing.

[0050] 2. Select the base model: The core architecture of the classifier is based on convolutional neural network (CNN). CNN has powerful feature extraction capabilities in the field of image processing and is suitable for extracting features of crop diseases and pests from images.

[0051] Choose a model that has been trained on a large-scale dataset as the base model. Such a model has learned rich feature representation capabilities and can be used as the initial model for new tasks to accelerate the training process and improve model performance.

[0052] 3. Freeze and fine-tune: In the pre-training phase, the convolutional layers of the base model are frozen, and only the newly added classifier is trained. The reason for this is that the convolutional layers of the base model have learned general feature representations on large-scale datasets, while the newly added classifier needs to be trained for specific crop pests and diseases.

[0053] After the classifier training is completed, unfreeze the convolutional layer of the base model and continue to train the entire model based on the collected data. This step is to make the entire model adaptable to the new task and data distribution.

[0054] 4. Model training and evaluation: The model is trained using the labeled data set. During the training process, the model parameters are continuously adjusted through the back propagation algorithm to minimize the value of the loss function.

[0055] The model is evaluated using evaluation indicators such as accuracy, recall, and F1 score. These indicators can fully reflect the performance of the model, including recognition accuracy, completeness, and comprehensive performance.

[0056] 5. Parameter adjustment and optimization: During the training process, parameter settings such as learning rate, batch size, number of iterations, etc. are constantly adjusted according to the performance of the model. The adjustment of these parameters is crucial to the convergence speed and final performance of the model.

[0057] Through continuous training and evaluation, the model converges. Convergence means that the model has learned enough feature representations to achieve stable and good performance on new data.

[0058] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A multimodal method based on CLIP to realize an intelligent crop recognition system, characterized in that: include: A data collection module, used to collect real-time images, text descriptions and environmental data of crops, wherein the text descriptions include descriptions of the types of crops and symptoms of pests and diseases; An image processing module is used to perform noise reduction on the real-time image, adjust the contrast and brightness of the real-time image to a set value, and distinguish the crop area from the background area in the real-time image based on image segmentation technology; The feature extraction module is used to extract features from real-time images and text descriptions based on the CLIP model, extract visual features from real-time images, extract semantic features from text descriptions, fuse environmental data with visual features, and generate multimodal feature vectors; The feature classification module is used to classify the multimodal feature vectors using a pre-trained classifier and identify the types of crops and the corresponding pests and diseases based on the extracted multimodal features; The result output module is used to display the recognition results to the user through a visual interface and feed back the information to the monitoring center.

2. The method of realizing intelligent crop identification system based on CLIP multimodal method according to claim 1, characterized in that: In the data acquisition module, the environmental data includes but is not limited to temperature, humidity, and soil nutrient concentration.

3. The method of realizing intelligent crop identification system based on CLIP multimodal method according to claim 1, characterized in that: In the image processing module, the noise reduction process is as follows: The real-time image is converted into a grayscale image, and the size of the grayscale image is adjusted to a set specification, and then the grayscale is wavelet transformed to decompose the grayscale image into several scales. The grayscale image is decomposed into several wavelet coefficients, and the wavelet coefficients represent the local features of the image at different scales. The wavelet coefficients of the high-frequency sub-band are threshold processed, and the noise coefficient is suppressed by the set threshold, and the signal coefficient is retained. After the threshold processing, the image is reconstructed based on the retained signal coefficient and wavelet coefficient, and the processed coefficient is converted back to the spatial domain through the inverse wavelet transform to obtain the denoised grayscale image and the real-time image.

4. The method of realizing intelligent crop identification system based on CLIP multimodal method according to claim 1, characterized in that: In the feature extraction module, the specific process of CLIP model extracting features is as follows: The CLIP model includes an image encoder and a text encoder. The image encoder is based on a ResNet network and converts an input real-time image into a visual feature vector and extracts visual features from the visual feature vector. The text encoder is based on a Transformer structure and is used to convert an input text description into a text feature vector and capture semantic features in the text from the text feature vector. The pre-paired visual feature vectors and text feature vectors are marked as positive sample pairs, and the unpaired ones are marked as negative sample pairs; the CLIP model embeds the visual feature vectors and text feature vectors into a shared semantic space, calculates the similarity of the visual feature vectors and the text feature vectors in the semantic space, and adjusts the model parameters according to the calculation results so that the similarity of the positive sample pairs approaches 1 and the similarity of the negative sample pairs approaches 0. The CLIP model uses a symmetric loss function, which consists of two parts of cross entropy. The corresponding loss is calculated from the row direction and the column direction respectively, and finally the average of the loss in the two directions is taken.

5. The method of realizing intelligent crop identification system based on CLIP multimodal method according to claim 1, characterized in that: For each real-time image, the model calculates the similarity between its visual feature vector and all text feature vectors, and converts all similarities into probability distributions through the softmax function. For each text description, the model calculates the similarity between its text feature vector and all visual feature vectors, and minimizes the inner product with irrelevant features. During the training process, the parameters are continuously adjusted to minimize the value of the loss function until convergence.

6. The method of realizing intelligent crop identification system based on CLIP multimodal method according to claim 1, characterized in that: The data acquisition module is also used to perform unit conversion and normalization processing on all environmental data. The normalization formula is x' = (x - min(x)) / (max(x) - min(x)), where x' represents the normalized data and x represents the original data.

7. The method of realizing intelligent crop identification system based on CLIP multimodal method according to claim 1, characterized in that: The data collection module is also used to perform deduplication processing on the text descriptions. For any pair of text descriptions with different dates but the same number of words, they are marked as the first text and the second text respectively, the occurrence frequencies of preset keywords in the text descriptions are collected respectively, the number of the top three keywords with the highest occurrence frequencies is collected, the number of occurrences of the top three keywords of the first text is marked as n1, n2 and n3, the number of occurrences of the top three keywords of the second text is marked as m1, m2 and m3, the three values ​​are used as the upper base, lower base and height of the trapezoid respectively, the first trapezoid and the second trapezoid are generated respectively, the area S1=(n1+n2)n3 / 2 of the first trapezoid is calculated, the area S2=(m1+m2)m3 / 2 of the first trapezoid is calculated, the first trapezoid and the second trapezoid are superimposed on a plane, the superposition state when the superposition area is the largest is selected, the area S3 of the superposition area under the current superposition state is obtained, the value of S3 / (S1+S2) is calculated, and the value is the shape similarity. Two text descriptions with a similarity greater than 90% are marked as duplicate texts, and the text description with the later date is deleted.

8. The method of realizing intelligent crop identification system based on CLIP multimodal method according to claim 1, characterized in that: The process of fusion feature in the feature extraction module is as follows: An encoder is used to extract features from environmental data, and the numerical features of environmental data are extracted based on the attention mechanism. The features are mapped into vector form in the multimodal space, and the extracted visual features and numerical features are weightedly fused. Different weights are assigned to the visual features and numerical features, and then the weighted sum is used to obtain the comprehensive features.

9. The method of realizing intelligent crop identification system based on CLIP multimodal method according to claim 1, characterized in that: The pre-training process of the classifier is: Label the image data, text data and environmental data, clean, denoise and standardize the collected data, use the convolutional neural network as the classifier, and select the model trained on a large-scale data set as the base model. Freeze the convolution layer of the base model and only train the newly added classifier. After the classifier training is completed, unfreeze the convolution layer of the base model and continue to train the entire model based on the collected data. The model is trained using labeled data sets and evaluated using evaluation metrics, including accuracy, recall, and F1 score. During the training process, the parameters are continuously adjusted to minimize the value of the loss function until the model converges.

Images