Land resource monitoring target detection method and system based on large model
Through the large-model-based land and resources monitoring target detection method, Vision Transformer and Adam optimizer are used to solve the problem of misidentification and false alarms of target detection in different environments, realizing zero-sample generalization and efficient detection, reducing costs and time expenditure.
Patent Information
- Application Number
- CN202411314337.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2024-12-13
AI Technical Summary
Existing object detection technology is prone to misidentification and false alarms when applied in different natural environments, resulting in a lot of time and manpower to make targeted model adjustments, which increases cost and complexity.
Using large-model-based land and resource monitoring target detection method, Vision Transformer and Adam optimizer, by cutting images, building object detection models and optimizing, reducing dependence on on-site data, providing zero-sample generalization capability, and reducing false positive errors Report.
The generalization ability of object detection in different environments is achieved, which significantly shortens the model training time and cost, reduces false positive errors, and improves the efficiency and accuracy of detection.
Smart Images
Figure CN119131367A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of target detection, and in particular to a land resources monitoring target detection method and system based on a large model. Background Art
[0002] Existing target detection technology can only generalize target detection on existing training data sets. When the same model needs to be applied in different environments, a large number of misidentifications will occur. Therefore, it is necessary to continue to collect a large number of data images of the current environment, re-annotate them and add them to the original training data set, and retrain the model to reduce false positives.
[0003] (1) For example, when the traditional firework recognition algorithm is applied to a new natural environment, it will mistakenly identify the fire clouds as flames because of their red color, irregular shape, and fluidity, and then issue an alarm.
[0004] (2) When the traditional algorithm for identifying floating objects on the water surface is applied to a new natural environment, it may mistakenly identify water birds flying over the water surface and aquatic plants drifting by as floating objects, thereby issuing an alarm.
[0005] (3) Under different natural environments, the situations that cause false alarms are unpredictable. When the same recognition model is to be applied on a large scale, traditional methods require a lot of time and manpower costs to adjust the model specifically for each environment. Summary of the invention
[0006] In order to solve the above-mentioned problems, the present invention provides a land resources monitoring target detection method and system based on a large model. Based on a multimodal large model, a new target detection method against false positives is used to reduce the dependence on a large amount of field data.
[0007] In the first aspect, the present invention provides a land resources monitoring target detection method based on a large model, which adopts the following technical solution: A land resources monitoring target detection method based on a large model, comprising: Get the target image; Performing cutting processing on the acquired target image; Construct a target detection model based on Vision Transformer, and use the target detection model to train the segmented target image to obtain a target detection model based on ViT-L. The Adam optimizer is used to optimize the ViT-L based object detection model; Use the optimized target detection model to perform target detection.
[0008] Furthermore, the acquired target image is subjected to a cutting process, including cutting the image into 224x224 images and then making them into a patch format, the patch size being 16*16, and constructing a fusion vector of the position matrix and the patch matrix according to the aforementioned process.
[0009] Furthermore, the construction of the target detection model based on Vision Transformer includes using the previous layer of the Transformer encoder of the ViT-L model as a feature extractor, and adding a fully connected layer and a detection head of yolov5 as the output layer of the target detection model.
[0010] Furthermore, the target detection model is used to train the cut target image, including using a feature extractor to extract features from the target image, and the output is recorded as , the fully connected layer is used to perform nonlinear transformation on the extracted features, and the output is recorded as , using yolov5's detection head Perform multi-scale prediction and obtain the prediction result, the output is recorded as The output format is (x, y, w, h, objectness, class probabilities, where x, y, w, h represent the center point position, width and height of the rectangular box, objectness represents the corresponding category label, and probabilities represents the corresponding confidence.
[0011] Furthermore, the Adam optimizer is used to optimize the ViT-L-based target detection model, including using a loss function to calculate the parameter loss of the true annotation box and the model prediction output, and the ViT-L-based target detection model is obtained by defining the average cross entropy loss and using the Adam optimizer to optimize the model until the model converges.
[0012] Furthermore, the loss function is used to calculate the parameter loss of the real annotation box and the model prediction output, including defining the bounding box regression loss MSE loss to measure the difference between the predicted bounding box and the real bounding box, wherein the MSEloss formula is: Where n is the number of samples, is the true value of the i-th sample, is the predicted value of the ith sample.
[0013] Furthermore, the use of the loss function to calculate the parameter loss of the true annotation box and the model prediction output also includes defining a classification loss Cross Entropy loss to measure the difference between the predicted category probability and the true category, wherein the Cross Entropy loss formula is: in, is the true label of sample i in category The value on is a binary variable. If the true category of the ith sample is the cth category, then =1, otherwise =0, is the sample predicted by the model Belongs to category The probability of represents the natural logarithm, is the number of samples in the dataset.
[0014] In the second aspect, a land resources monitoring target detection system based on a large model includes: The data acquisition module is configured to acquire a target image; The preprocessing module is configured to perform a cutting process on the acquired target image; The model training module is configured to build a target detection model based on Vision Transformer, and use the target detection model to train the segmented target image to obtain a target detection model based on ViT-L; The model optimization module is configured to optimize the ViT-L based object detection model using the Adam optimizer; The detection module is configured to perform target detection using the optimized target detection model.
[0015] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, for example, a large-model-based land and resources monitoring target detection method.
[0016] In a fourth aspect, the present invention provides a terminal device including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the method for detecting a land and resources monitoring target based on a large model.
[0017] In summary, the present invention has the following beneficial technical effects: The present invention can solve the target detection problem in the land and resources monitoring scenario through the above technical solution. By using the strong context learning and reasoning ability of the large model, the problem of poor generalization ability in the traditional solution can be solved, and a zero-sample generalization ability is provided. At the same time, by using the positive sample feature library and the negative sample feature library, the retraining process of false positive data is omitted, and the time cost and data cost are reduced.
[0018] The present invention does not require a large amount of real-world data to retrain the model during scene migration like traditional deep learning methods. Instead, it utilizes the strong feature acquisition and contextual reasoning capabilities of large models and only requires one labeled picture to solve the false positive problem, greatly shortening the technical path and significantly reducing the time and cost of training the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 is a schematic diagram of training based on the ViT-L target detection model according to Example 1 of the present invention; Figure 2 It is a schematic diagram of the feature database generation process of Example 1 of the present invention. DETAILED DESCRIPTION The present invention is further described in detail below in conjunction with the accompanying drawings.
[0020] Example 1 Reference Figure 1 , a land resources monitoring target detection method based on a large model in this embodiment includes: Get the target image; Performing cutting processing on the acquired target image; Construct a target detection model based on Vision Transformer, and use the target detection model to train the segmented target image to obtain a target detection model based on ViT-L. The Adam optimizer is used to optimize the ViT-L based object detection model; Use the optimized target detection model to perform target detection.
[0021] Specifically, the following steps are included: S1. Acquire the target image; The image data comes from the user's real-time video stream, which is usually pulled using the RTSP protocol.
[0022] S2. Perform cutting processing on the acquired target image; Among them, in Reshape PyTorch, reshape() is a very useful function that allows you to change the shape of a tensor while keeping the data unchanged. This is very common in deep learning, especially when you need to adjust the size of tensors so that they can be passed to different layers of the model.
[0023] (1) First, extract a 14*14 sliding window and split the image along the width and height of the image. Next, adjust the dimension of the tensor to flatten the patch list (16*16). Finally, add position embedding and perform linear transformation on each patch to meet the input requirements of ViT. Make patches into (batch, num_patches, patch_size) shape.
[0024] (2) Post-add position embedding: Create a position embedding matrix, initialize it using the random method, and add it to the patches. (The size of the position embedding is usually the same as the size of the flattened patches, that is, the dimension of each position embedding vector should be consistent with the dimension of each patch) (3) Apply linear transformation: Use a linear transformation layer to adjust the dimensions of patches.
[0025] The cropping operation is a Crop operation that uses a sliding window to cut out a 224x224 pixel sub-image from the original image. Next, the cropped image is divided into multiple small square areas, called "patches". For example, if a 16x16 pixel patch is used, the 224x224 image is divided into =14×14=196 patches. Each patch is a 16×16×3 three-dimensional array. For ease of processing, each patch is flattened into a one-dimensional vector. This means that each patch becomes a vector of length 16×16×3=768. After the patches are flattened into vectors, the vectors pass through a linear projection layer. This layer maps the flattened vectors to a new feature space whose dimensions are determined by the model designer. Assuming the output dimension of the linear projection is d, then for each patch we have the following relationship: P is the side length of the patch; C is the number of color channels (3 in this case); and d is the dimension of the vector obtained after linear projection.
[0026] In order to preserve the position information, positional encoding is added to the patch vector. This is because the position information of the sequence is very important in the transformer architecture, and the original patch vector itself does not contain position information.
[0027] Position encoding: It is used to embed position information in the patch vector, because the patch vector itself does not carry position information. The commonly used position encoding method in ViT is the position encoding based on sine and cosine functions.
[0028] The position encoding is usually a matrix that matches the number of patches, with each row corresponding to the position information of a patch. If the image is divided into N patches, the position encoding matrix is an N×d matrix.
[0029] Where N is the number of patches and d is the dimension of each patch vector.
[0030] The calculation formula for each item of the position encoding is as follows: Combination of patch vector and position encoding: Finally, each patch vector is added to its corresponding position code to form a new vector with position information. The patch vector pi plus the position encoding The resulting vector is: .
[0031] S3. Construct a target detection model based on Vision Transformer, and use the target detection model to train the segmented target image to obtain a target detection model based on ViT-L; Among them, ViT-L is a larger vision Transformer model with 24 encoder layers, 1024-dimensional hidden layers per layer, 16 attention heads, and about 300 million parameters. It takes 16×16 image blocks as input and uses position embedding to preserve spatial information. By building a target detection model based on Vision Transformer, the penultimate layer of ViT-L (that is, the layer before the last layer of Transformer encoder) is used as the feature extractor. A fully connected layer and the detection head of yolov5 are added as the output layer of the target detector.
[0032] Build the detection head. The detection head consists of 5 convolutional layers, which are used to predict bounding boxes and class probabilities from the fused feature maps. Each grid cell on each feature map predicts a certain number of bounding boxes and their confidence scores and class probabilities. The output format is (x, y, w, h, objectness, class probabilities), where (x, y) is the coordinates of the center of the bounding box relative to the upper left corner of the grid cell, (w, h) is the width and height of the bounding box, objectness is the probability that the bounding box contains an object, and class probabilities is the probability distribution of each class.
[0033] Specifically, (1) Feature extraction stage: The penultimate Transformer encoder layer in the ViT-L model is used as a feature extractor, which contains rich contextual information obtained after multi-scale analysis of the input image. The output is recorded as (2) Feature conversion stage: A fully connected layer is added based on the feature extraction stage. The input is The function of this fully connected layer is to perform nonlinear transformation on the extracted features to meet the needs of subsequent detection tasks. The fully connected layer can further compress the feature space, making it more suitable for detection tasks and helping to reduce model complexity. The output of this layer is recorded as .
[0034] (3) Detection output stage: The input of this layer is .
[0035] (3.1) Detection head architecture: Path Aggregation Network (PANet): The detection head of YOLOv5 usually adopts a structure similar to PANet, which generates feature maps of different scales through upsampling and feature fusion to adapt to the detection of objects of different sizes.
[0036] (3.2) Prediction module: Each generated feature map passes through a prediction module, which contains several convolutional layers to refine the features and predict the location and class probability of the bounding box.
[0037] (3.3) Multi-scale prediction: Prediction is performed simultaneously on feature maps of multiple scales, which helps improve the detection performance of small and large objects. For each scale of the feature map, the detection head outputs a set of bounding boxes and corresponding class probabilities. The output bounding box includes the center point coordinates, width, and height.
[0038] The category probability represents the possibility of belonging to each category and is normalized using the Softmax function.
[0039] (3.4) Non-maximum suppression (NMS): To remove duplicate detection results, the NMS algorithm is applied, which sorts the predicted bounding boxes according to their confidence scores and removes bounding boxes with high overlap.
[0040] Final output: The final output of the model The output format is (x, y, w, h, objectness, classprobabilities, where (x, y, w, h represent the center point position and width and height of the rectangular box, objectness represents the corresponding category label, and probabilities represents the corresponding confidence.
[0041] S4. Use Adam optimizer to optimize the ViT-L based object detection model; Among them, the loss function is used to calculate the parameter loss of the real annotation box and the model prediction output. By defining the average cross entropy loss and using the Adam optimizer to optimize the model until the model converges, the object detection model based on ViT-L is obtained.
[0042] Real annotation box: It is the data pre-annotation. The process is to use manual annotation to annotate each pixel of the "target to be detected" in a picture to obtain an annotation box, which contains four parameters x, y, w, and h, which represent the center point coordinates (x, y) of the bounding box and the width w and height h of the box respectively. Model prediction output: (1) Define the loss function, including bounding box regression loss and classification loss. Bounding box regression loss is used to measure the difference between the predicted bounding box and the true bounding box. Here we choose MSE loss.
[0043] The formula for MSE LOSS is: n is the number of samples.
[0044] is the true value of the i-th sample.
[0045] is the predicted value of the ith sample.
[0046] Classification loss is used to measure the difference between the predicted category probability and the true category. Here we choose to use the formula of CrossEntropy loss: is the true label of sample i in category The value on (if it is one-hot encoding, then is 0 or 1).
[0047] is the sample predicted by the model Belongs to category probability.
[0048] Represents the natural logarithm.
[0049] is the number of samples in the dataset.
[0050] (2) Adam optimization process (2.1) Initialize the optimizer: Use the Adam optimizer to initialize the optimization process, setting appropriate learning rate = 0.001, beta1 = 0.9, beta2 = 0.999 and epsilon = 1e-8.
[0051] (2.2) Training loop: For each epoch: (2.2.1) The input image passes through the ViT-L feature extractor to obtain a feature map. The feature map is transformed through a fully connected layer. The transformed features are passed through the YOLOv5 detection head to obtain the predicted bounding box and category probability.
[0052] (2.2.2) Calculate the loss function.
[0053] (2.2.3) Use the Adam optimizer for backpropagation to update the model parameters.
[0054] (2.2.4) Clear the gradient.
[0055] (2.3) When the performance of the model on the validation set no longer improves, stop training. Or when the maximum number of iterations (2000) is reached, the training is automatically terminated.
[0056] S5. Use the optimized target detection model to perform target detection.
[0057] Figure 2 It shows how to extract features through the ViT-L model and generate a feature database containing positive and negative samples through manual verification. This database can be used for feature comparison of positive and negative samples in the actual application stage.
[0058] Select 5 positive samples: 1. First, select 5 known positive samples from the dataset. These positive samples are samples that are known to belong to a specific category.
[0059] 2. Through the second-to-last layer of ViT-L data: Input these 5 positive samples into the ViT-L (Vision TransformerLarge) model and extract their outputs in the second-to-last layer of the model as feature vectors.
[0060] The extracted feature vector is used as the feature representation of the positive sample and stored in the local database.
[0061] 3. Actual images to be detected: For the images that actually need to be detected, they are also processed by the object detection model based on ViT-L. Extract the second-to-last layer data of ViT-L as the feature vector: Extract the output of the second-to-last layer of the ViT-L model from the actual images to be detected as the feature vector.
[0062] 4. Negative samples verified by manual verification: These feature vectors extracted from the actual images to be tested are manually verified to be negative samples (i.e., samples that do not belong to the positive sample category).
[0063] 5. Aggregate to generate a database containing positive samples and negative samples: Aggregate the manually verified negative sample feature vectors with the previously stored positive sample feature vectors to generate a feature database containing positive samples and negative samples.
[0064] Example 2 (1) Select an image of the target to be detected, input it into the ViT-L model, save the output of the second-to-last layer of ViT-L, and reshape it into a one-dimensional vector as a reference vector for subsequent similarity calculation.
[0065] The above processing is performed for each type of target to be detected and saved to the local database according to different classifications.
[0066] (2) Access the land and resources monitoring video stream to be monitored, extract the frames into single images, and input them into the target detection model based on ViT-L. Obtain the target detection frame. Obtain the detected category. Go to step (3).
[0067] (3) Save the output of the penultimate layer of ViT-L and reshape it into a one-dimensional vector as the comparison vector for subsequent similarity calculation.
[0068] (4) VisionTransformer extracts data features of new images.
[0069] The features extracted this time and the classification category of the detection target are matched with the corresponding category features saved in step (1) for similarity, and the similarity calculation method adopts the weighted cosine similarity method.
[0070] The output vector of the penultimate layer of the ViT-L model has the following characteristics: ① Multimodal information: The penultimate layer output of iT contains rich multi-scale features, reflecting the information at different levels of the image. These features include not only local texture information, but also global spatial layout and context information.
[0071] ② Position information: Since ViT uses position embeddings to maintain spatial information, the penultimate layer output vector contains position information. This means that each element in the vector corresponds to a specific position in the image.
[0072] ③Semantic information: The output vector of the penultimate layer of ViT contains strong semantic information, which helps to identify object categories and relationships in the image.
[0073] Therefore, the simple weighted cosine similarity calculation method cannot fully utilize its features, and the similarity calculated by it has limitations.
[0074] This embodiment proposes a new similarity calculation method: weighted cosine similarity.
[0075] ① Calculate position weights: Calculate a weight for each position (i.e. each patch) to reflect its importance in the image. The weight can be determined based on factors such as the centrality of the position in the image, edge effects, etc. Assign higher weights to patches in the center of the image because they usually contain richer information.
[0076] ② Calculate semantic weights: Calculate a weight for each dimension (i.e., each element in the feature vector) to reflect its importance in semantic expression. The weights are determined by analyzing the variability and contribution of each dimension in the training dataset. Use principal component analysis (PCA) or feature importance analysis to determine which dimensions are more important.
[0077] ③ Calculate weighted cosine similarity: Use position weight and semantic weight to calculate the weighted cosine similarity between two vectors.
[0078] The final calculation formula is as follows: Where n is the dimension of the vector, wp(i)wp(i) is the weight of position ii, ws(i)ws(i) is the semantic weight of the ii-th dimension, and AiAi and BiBi are the values of the two vectors in the ii-th dimension respectively.
[0079] (5) Determine the cosine similarity between the feature and the target to be detected in the local feature database. If the similarity is greater than 90%, it is determined to be the target to be detected. If the similarity is less than 90%, no processing is performed.
[0080] (6) Determine the cosine similarity between the feature and the feature of the false positive target in the local feature database. If the similarity is less than 50%, it is determined to be a target to be detected. If the similarity is greater than 50%, no processing is performed.
[0081] (7) The submitted images will be reviewed by the reviewer. If the review result is correct, no action will be taken. (8) If the verification result is an error, go to step (2) to extract the features of the target information of this alarm as false positive features and save them in the local database.
[0082] (9) Go to step (2).
[0083] Example 3 This embodiment provides a land resources monitoring target detection system based on a large model, including: The data acquisition module is configured to acquire a target image; The preprocessing module is configured to perform a cutting process on the acquired target image; The model training module is configured to build a target detection model based on Vision Transformer, and use the target detection model to train the segmented target image to obtain a target detection model based on ViT-L; The model optimization module is configured to optimize the ViT-L based object detection model using the Adam optimizer; The detection module is configured to perform target detection using the optimized target detection model.
[0084] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the method described.
[0085] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; and the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the described method.
[0086] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A land resources monitoring target detection method based on a large model, It is characterized in that include: Get the target image; Performing cutting processing on the acquired target image; Construct a target detection model based on Vision Transformer, and use the target detection model to train the segmented target image to obtain a target detection model based on ViT-L. The Adam optimizer is used to optimize the ViT-L based object detection model; Use the optimized target detection model to perform target detection.
2. According to claim 1, a land resources monitoring target detection method based on a large model, It is characterized in that The obtained target image is cut into pieces, including cutting the image into 224x224 images and then making them into patch formats, the patch size is 16*16, and the total size is 14*14 patches.
3. According to claim 2, a large model-based land resources monitoring target detection method, It is characterized in that The target detection model based on Vision Transformer is constructed, including using the previous layer of the Transformer encoder of the ViT-L model as a feature extractor, and adding a fully connected layer and a detection head of yolov5 as the output layer of the target detection model.
4. According to claim 3, a large model-based land resources monitoring target detection method, It is characterized in that The target detection model is used to train the cut target image, including extracting features from the target image using a feature extractor, and the output is recorded as , the fully connected layer is used to perform nonlinear transformation on the extracted features, and the output is recorded as , using yolov5's detection head Perform multi-scale prediction and obtain the prediction result, the output is recorded as The output format is (x, y, w, h, objectness, class probabilities, where x, y, w, h represent the center point position, width and height of the rectangular box, objectness represents the corresponding category label, and probabilities represents the corresponding confidence.
5. According to the large model-based land resources monitoring target detection method of claim 4, It is characterized in that The Adam optimizer is used to optimize the ViT-L-based target detection model, including using a loss function to calculate the parameter loss of the true annotation box and the model prediction output, and the ViT-L-based target detection model is obtained by defining an average cross entropy loss and optimizing the model using the Adam optimizer until the model converges.
6. According to claim 5, a large model-based land resources monitoring target detection method, It is characterized in that The loss function is used to calculate the parameter loss of the real annotation box and the model prediction output, including defining the bounding box regression loss MSE loss to measure the difference between the predicted bounding box and the real bounding box, where the MSE loss formula is: Where n is the number of samples, is the true value of the i-th sample, is the predicted value of the ith sample.
7. According to claim 6, a large model-based land resources monitoring target detection method, It is characterized in that The loss function is used to calculate the parameter loss of the real annotation box and the model prediction output, and also includes defining the classification loss Cross Entropy loss to measure the difference between the predicted category probability and the real category, wherein the Cross Entropy loss formula is: in, is the true label of sample i in category The value on is a binary variable. If the true category of the ith sample is the cth category, then =1, otherwise =0, is the sample predicted by the model Belongs to category The probability of represents the natural logarithm, is the number of samples in the dataset.
8. A land resources monitoring target detection system based on a large model, It is characterized in that include: The data acquisition module is configured to acquire a target image; The preprocessing module is configured to perform a cutting process on the acquired target image; The model training module is configured to build a target detection model based on Vision Transformer, and use the target detection model to train the segmented target image to obtain a target detection model based on ViT-L; The model optimization module is configured to optimize the ViT-L based object detection model using the Adam optimizer; The detection module is configured to perform target detection using the optimized target detection model.
9. A computer-readable storage medium having a plurality of instructions stored therein, It is characterized in that The instructions are suitable for being loaded by a processor of a terminal device and executing a land and resources monitoring target detection method based on a large model as described in claim 1.
10. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions. It is characterized in that The instructions are suitable for being loaded by a processor and executing a land and resources monitoring target detection method based on a large model as described in claim 1.