Low-altitude ground feature recognition sample automatic labeling method and system applying SAM fine tuning

Abstract

The invention discloses a low-altitude ground feature recognition sample automatic labeling method and system applying SAM fine tuning, and belongs to the technical field of agricultural remote sensing technologies and unmanned aerial vehicles. According to the method, the trained model branch is connected to the SAM main body network, the initial masks are extracted from the samples, the initial masks conforming to the target category features are obtained through screening and are automatically labeled, and in the scene that large-scale remote sensing data is to be labeled, the processable image data size and the labeled sample number are greatly increased; through the arrangement of the model branch, different model branch parameters are obtained for different to-be-identified or to-be-labeled targets, and the identification and classification of various types of crops and agricultural ground features can be realized; the obtained model fine tuning parameters and the SAM backbone network are separately stored, and a plug-and-play structural design is realized, so that the method can flexibly adapt to agricultural remote sensing data in different regions and different seasons, the recognition performance and generalization ability are stable, and the application scene is wide.

Description

TECHNICAL FIELD

[0001] The application relates to the fields of agricultural remote sensing and unmanned aerial vehicle technology, and in particular to a low-altitude ground object identification sample automatic labeling method and system applying SAM fine tuning. BACKGROUND

[0002] In the field of unmanned aerial vehicle agricultural remote sensing, ground object identification is often realized through an image recognition model, and in the identification process, the construction and training of the image recognition model are involved.

[0003] In the existing image recognition model training process, high-quality samples required by the image recognition model are often manually outlined, and complex targets such as farmland boundaries, crop types and disease areas are labeled to realize fine segmentation of ground objects.

[0004] In actual application, the above-mentioned manual implementation method not only causes high labeling cost and low efficiency, but also cannot further realize labeling of a large number of samples, and instead causes a shortage of labeled samples, thereby restricting the application of a deep learning model. SUMMARY

[0005] In order to solve the problems in the prior art, the application embodiment provides a low-altitude ground object identification sample automatic labeling method and system applying SAM fine tuning. The technical solution is as follows: On one hand, a low-altitude ground object intelligent identification sample automatic labeling method applying SAM fine tuning is provided, and the method comprises the following steps: a model LoRA fine tuning framework based on a SAM backbone network is built, and the SAM backbone network is separated from fine tuning parameters of a fine tuning model; a current target corresponding sample is collected and made by an unmanned aerial vehicle, and different ground object target samples are classified and stored; the SAM backbone network is frozen, and an incremental training is performed on the model branch according to the sample, and the obtained model fine tuning parameters are stored separately from the SAM backbone network, and different ground object target model branch parameters are classified and stored; the trained model branch is connected to the SAM backbone network, and an initial mask is generated based on the complete network and the sample; features are calculated by an image encoder of a SAM vision model for an image region covered by the initial mask, and related features are obtained; similarity calculation is performed on the related features and existing feature vectors in a prototype memory bank, and an initial mask conforming to a target category feature is screened and obtained, and automatic labeling is completed.

[0006] Optionally, the method further comprises the following steps: From multiple ground object targets, the next target continues to be trained to obtain model fine-tuning parameters corresponding to the next target, until the training of multiple targets is completed; For different targets, the corresponding model fine-tuning parameters are matched, and the step of connecting the trained model branch to the SAM main network and extracting the initial mask of the sample is performed.

[0007] Optionally, the model LoRA fine-tuning framework based on the SAM backbone network comprises: Corresponding to the self-attention calculation network layer of the SAM backbone network, a branch network layer is added; Corresponding to the output neural network layer of the SAM backbone network, a low-rank feature extraction network layer is added.

[0008] Optionally, the step of connecting the trained model branch to the SAM backbone network and generating an initial mask for the sample based on a complete network comprises: Connecting the trained model branch to the SAM backbone network to obtain a fine-tuned SAM model corresponding to the current target; Extracting the initial mask corresponding to the current target from the sample through the fine-tuned SAM model; Calculating the relevant features corresponding to the image covered by the initial mask; Continuing to obtain the initial mask of the next target until the initial masks and relevant features of all targets are obtained.

[0009] Optionally, before the step of calculating the similarity between the relevant features and the existing feature vectors in the prototype memory bank and screening the initial mask that meets the target category characteristics to complete automatic labeling, the method further comprises: Calculating a plurality of features corresponding to the sample that has been accurately labeled; Obtaining a plurality of feature vectors corresponding to the plurality of features; Obtaining the prototype memory bank according to the plurality of feature vectors.

[0010] Optionally, the step of calculating the similarity between the relevant features and the existing feature vectors in the prototype memory bank and screening the initial mask that meets the target category characteristics to complete automatic labeling comprises: Setting a prototype clustering algorithm; Calculating the similarity between the relevant features and the prototype memory bank through the prototype clustering algorithm, and screening the relevant features with a similarity that meets a threshold value; Setting the initial mask corresponding to the relevant features with a similarity that meets the threshold value as a pseudo label.

[0011] Optionally, the method further comprises: Verify the pseudo label, and set the sample with successful verification as an effective labeled sample; Sample re-collection is performed on the effective labeled sample.

[0012] Optionally, the connecting of the trained model branch to the SAM backbone network and the initial mask generation based on the complete network further include: Obtaining the initial weight of the SAM backbone network, and obtaining the training weight of the branch network through LoRA fine-tuning; According to the initial weight and the training weight, the weight of the fine-tuned SAM model is obtained; Through the fine-tuned SAM model, the initial mask corresponding to the current target is extracted.

[0013] On the other hand, a low-altitude ground object intelligent identification sample automatic labeling system applying SAM fine-tuning is also provided, which includes a UAV data acquisition system, a parameter plug-and-play structure, and an automatic labeling model, wherein: The automatic labeling model is used to build a model LoRA fine-tuning framework based on a SAM backbone network, and the SAM backbone network and the fine-tuning model are separated from the fine-tuning parameters; The UAV data acquisition system is used to acquire and produce samples corresponding to the current target; The automatic labeling model is used to: Store the samples and store the samples corresponding to different ground object targets in categories; Freeze the SAM backbone network, and perform incremental training on the model branch according to the samples, store the obtained model fine-tuning parameters separately from the SAM backbone network, and store the model branch parameters of different ground object targets in categories; The parameter plug-and-play structure is used to match and adapt to multi-class ground object target labeling generation; The automatic labeling model is used to: Connect the trained model branch to the SAM backbone network, and generate an initial mask based on the complete network; Through the image encoder of the SAM visual model, the feature of the image region covered by the initial mask is calculated to obtain the related feature; The related feature and the prototype memory bank are calculated for similarity, and the initial mask that meets the target class feature is screened to complete automatic labeling.

[0014] Optionally, the automatic labeling model is used to: Corresponding to the network layer of the SAM backbone network, a branch network layer is added; The output neural network layer corresponding to the SAM backbone network corresponds to an added low-rank feature extraction network layer.

[0015] The technical solution provided by the embodiment of the application has the following beneficial effects: 1. By connecting the trained model branch to the SAM main network, the initial mask of the sample is extracted, and the similarity between the relevant features and the prototype memory library is calculated to screen the initial mask of the target category feature for automatic labeling. In the scenario of large-scale remote sensing data to be labeled, the method provided by the embodiment of the application realizes automatic labeling through pseudo-label generation, which improves the efficiency compared with manual labeling, and realizes a substantial increase in the amount of processable image data and the number of labeled samples.

[0016] 2. By setting the model branch, different model branch parameters are obtained for different to-be-identified or to-be-labeled targets, which can realize accurate identification and classification of various types of crops and agricultural ground objects, and has good cross-region migration ability.

[0017] 3. By separating the obtained model fine-tuning parameters from the SAM backbone network for storage, a plug-and-play structure design is realized, so that the method can flexibly adapt to agricultural remote sensing data in different regions and different seasons, maintain stable identification performance and generalization ability, and be applied to more extensive agricultural application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 is a flowchart of a low-altitude ground object identification sample automatic labeling method using SAM fine-tuning provided by the embodiment of the application; Figure 2 is a flowchart of a low-altitude ground object identification sample automatic labeling method using SAM fine-tuning provided by the embodiment of the application; Figure 3 is a flowchart of a low-altitude ground object identification sample automatic labeling method using SAM fine-tuning provided by the embodiment of the application; Figure 4 is a schematic diagram of a low-altitude ground object identification sample automatic labeling system using SAM fine-tuning provided by the embodiment of the application. DETAILED DESCRIPTION

[0020] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0021] It should be noted that the low-altitude ground object intelligent identification sample automatic labeling method applying SAM fine-tuning described in the embodiments of the present application is mainly used for fine-tuning of a SAM visual large model to adapt to complex scenes in a target identification process (especially an agricultural ground object target identification process in unmanned aerial vehicle remote sensing). In actual application, the method described in the embodiments of the present application can also be applied to other visual large models, especially visual large models used for identifying high-resolution image data. The above SAM visual large model is only exemplary, and the embodiments of the present application do not limit the specific visual large model.

[0022] For the convenience of description, the embodiments of the present application take a single target, i.e., a current target, as an example for description. In actual application, a target identification process often involves a multi-target identification scene. The multi-target identification process is different from a single-target identification process in terms of samples, and the principle is the same as that of the single-target process.

[0023] In actual application, the low-altitude ground object intelligent identification sample automatic labeling method applying SAM fine-tuning provided by the embodiments of the present application adjusts a traditional artificial labeling mode to computer automatic labeling in combination with unmanned aerial vehicle high-resolution image data and fine-tuning of a SAM visual large model, so as to improve the accumulation speed of high-quality agricultural ground object samples.

[0024] Referring to Figure 1 The low-altitude ground object intelligent identification sample automatic labeling method applying SAM fine-tuning is provided, and the method comprises the following steps: 101. A model LoRA fine-tuning framework based on a SAM backbone network is built, and the SAM backbone network is separated from fine-tuning parameters of a fine-tuning model; 102. A sample corresponding to a current target is collected and made by an unmanned aerial vehicle, and different ground object target samples are classified and stored; 103. The SAM backbone network is frozen, and an incremental training is performed on a model branch according to the sample. The obtained model fine-tuning parameters are stored separately from the SAM backbone network, and different ground object target model branch parameters are classified and stored; 104. The trained model branch is connected to the SAM backbone network, and an initial mask is generated based on the complete network and the sample; 105. Using the image encoder of the SAM visual model, feature calculation is performed on the image region covered by the initial mask to obtain relevant features; 106. Calculate the similarity between the relevant features and the existing feature vectors in the prototype memory, select the initial mask that matches the target category features, and complete the automatic labeling.

[0025] Optionally, in practical applications, the following steps may be included before step 101: Selecting a large SAM visual model corresponding to the current scene can be achieved by selecting a validation sample corresponding to the current scene, inputting it into multiple large SAM visual models, and verifying the recognition results.

[0026] Alternatively, pre-trained samples corresponding to the current scene can be collected, and the SAM visual large model can be trained using the pre-trained samples to obtain pre-trained samples corresponding to the current scene.

[0027] It should be noted that in the above steps, the scenes can be distinguished by the different targets they contain, such as different types and quantities of crops or plants, which can be divided into different scenes.

[0028] Optionally, the method also includes: From multiple ground targets, the training continues for the next target, obtaining the model fine-tuning parameters corresponding to the next target, until the training of multiple targets is completed; For different objectives, the corresponding model fine-tuning parameters are matched, and the steps of connecting the trained model branches to the SAM main network and extracting the initial mask from the samples are performed.

[0029] It should be noted that the above process mainly applies to scenarios where the sample contains a single target (i.e., the current target). In practical applications, the sample can also contain multiple targets, and these multiple targets can be identified simultaneously. In scenarios where the sample contains multiple targets, it is not necessary to continue training for the next target to obtain the model fine-tuning parameters corresponding to the next target, until the training of multiple targets is completed.

[0030] In practical applications, identifying a single target and generating corresponding model fine-tuning parameters for different targets is more accurate than identifying multiple targets and generating corresponding model fine-tuning parameters. However, it also takes longer in practice. Identifying multiple targets and generating corresponding model fine-tuning parameters, although less accurate, takes less time. In practical applications, the choice between the two methods can be made based on the required accuracy and the number of targets to be identified.

[0031] Optionally, the model branches of the SAM backbone network set in step 101 include: Wherein, the branch network layer and the low-rank feature extraction network layer are determined through the weight matrix of the SAM visual large model; Corresponding to the self-attention calculation network layer of the SAM backbone network, a branch network layer is added; in actual application, the LoRA (Low-Rank Adaptation) rank in the above process can be determined by the target quantity in the sample, such as setting a target quantity threshold, if the target quantity in the sample is greater than or equal to the target quantity threshold, the LoRA rank is set to 64, and if the target quantity in the sample is less than the target quantity threshold, the LoRA rank is set to 32, and the specific target quantity threshold is not limited in the embodiment of the application.

[0032] Corresponding to the output neural network layer of the SAM backbone network, a low-rank feature extraction network layer is added; The output neural network layer can be a multi-head self-attention layer in actual application, and the specific low-rank feature extraction network layer and the corresponding rank are not limited in the embodiment of the application.

[0033] Optionally, the process of collecting and making the sample corresponding to the current target by the unmanned aerial vehicle in step 102 can be specifically as follows: Through the unmanned aerial vehicle, unmanned aerial vehicle orthographic image data of a plurality of regions is obtained, the plurality of regions at least including the current target; Radiometric correction and geometric correction are performed on the plurality of unmanned aerial vehicle orthographic image data, and the radiometric calibration, atmospheric correction, solar elevation angle and terrain correction processes in the above radiometric correction process are not limited in the embodiment of the application; The plurality of unmanned aerial vehicle orthographic image data is cut (such as cutting with a target size of 1024*1024); A plurality of targets in the cut image data are accurately labeled to obtain a plurality of samples, wherein the accurate labeling process can be realized by an artificial standard method.

[0034] Optionally, the process of freezing the SAM backbone network and incrementally training the model branch according to the sample to separate and store the obtained model fine-tuning parameters from the SAM backbone network in step 103 can be as follows: The SAM backbone network is frozen; The SAM visual large model containing the SAM backbone network and the branch model is incrementally trained through a plurality of samples, and model fine-tuning parameters are obtained; in the training process, the current target in the sample can be trained, such as only training the current target as rice in the sample containing agricultural ground object targets such as rice, rape and plastic greenhouse, to obtain model fine-tuning parameters corresponding to the rice; The training on the next target is continued to obtain the model fine-tuning parameters corresponding to the next target. For example, after the training on the rice is completed, the training on the rape is continued to obtain the model fine-tuning parameters corresponding to the rape, and the training on the agricultural surface objects such as the rice, the rape and the plastic greenhouse is completed.

[0035] Optionally, in the training process, the multiple targets in the sample can also be trained. For example, the targets such as the rice, the rape and the plastic greenhouse in the sample containing the agricultural surface objects such as the rice, the rape and the plastic greenhouse are trained to obtain the model fine-tuning parameters corresponding to the scene. Since the types of the agricultural surface objects contained in some specific scenes are relatively fixed, the training on the multiple targets in the sample can realize the labeling of the fixed types of the agricultural surface objects in the specific scenes. Or, when the target to be labeled or recognized is a specific multiple target, the multiple targets in the sample can be trained, and other agricultural surface objects in the sample can be ignored.

[0036] The obtained model fine-tuning parameters are stored separately from the SAM backbone network to realize the plug-and-play structure of the fine-tuning structure, facilitate the reuse of the backbone network and the adjustment of the recognition target, realize the flexible adjustment and replacement of the model inference, further realize the lightweight parameter storage, and reduce the resource occupation.

[0037] The model fine-tuning parameters of the SAM visual large model backbone network are frozen and do not participate in the training, and only the model branch is incrementally trained, so that the incrementally training of the model branch does not affect the SAM backbone network, and the general image feature extraction capability of the original SAM visual large model is retained.

[0038] Optionally, step 104 connects the trained model branch to the SAM backbone network, and performs initial mask generation on the sample based on the complete network, including: The trained model branch is connected to the SAM main network to obtain a fine-tuning SAM model corresponding to the current target; The above sample is extracted by the fine-tuning SAM model to extract an initial mask corresponding to the current target; The initial mask of the next target is continuously obtained until the initial masks of all targets are obtained.

[0039] Optionally, before step 105 performs similarity calculation on the related features and the prototype memory bank to screen the initial mask meeting the target category features, the method further includes: The multiple features corresponding to the accurately labeled sample are calculated; Wherein, for the current target, the process can be to input the sample obtained in step 102 into the image encoder in the SAM visual large model to obtain a plurality of features of the current target, and present them in the mask feature map; it should be noted that in this step, the image encoder of the SAM visual large model, the basic feature extractor of which is ViT, and the core of which is a self-attention layer; According to the plurality of features, a plurality of feature vectors corresponding thereto are obtained; Specifically, after obtaining a plurality of feature vectors from a single sample, the next sample is continuously inputted, a plurality of features are continuously obtained, and a plurality of feature vectors are obtained; According to the plurality of feature vectors, a prototype memory bank is obtained, which can also be updated on the basis of the existing prototype memory bank according to the plurality of feature vectors; wherein the prototype memory bank contains a plurality of feature vectors.

[0040] Optionally, the similarity calculation of the related features and the prototype memory bank in step 106 to filter the initial mask of the features meeting the target category includes: The prototype clustering algorithm is set, and the specific prototype clustering algorithm is not limited in the embodiment of the application.

[0041] The similarity calculation of the related features and the prototype memory bank is performed through the prototype clustering algorithm, and the related features whose similarity meets the threshold are filtered; the process can be: The plurality of vectors corresponding to the related features are calculated; The related features and the plurality of features are matched and calculated to obtain a plurality of matching vectors; The similarity calculation of the plurality of vectors, the plurality of matching vectors, and the plurality of feature vectors (or a plurality of prototypes) in the prototype memory bank is performed through the prototype clustering algorithm; According to the output result of the prototype clustering algorithm, the plurality of vectors and the plurality of matching vectors whose similarity meets the threshold are filtered; The related features corresponding to the plurality of vectors and the plurality of matching vectors are obtained; The initial mask corresponding to the related features is set as a pseudo label.

[0042] Wherein, the prototype memory bank is obtained by fusing the prototype network and the memory bank; The process described in steps 104 to 105 above can also refer to Figure 2 As shown, through the above steps, the initial mask and the corresponding pseudo label of the sample can be generated, and then step: Combined with the confidence threshold screening and the multi-temporal image cross verification, low-quality pseudo labels are filtered out; Optionally, the method further includes: Verify the pseudo label, and set the sample with successful verification as an effective labeled sample; The effective labeled sample is subjected to sample collection.

[0043] After the sample with successful verification is corrected by a small amount of manual correction, the training set can be iteratively expanded to form a closed loop of "fine-tuning-pseudo labeling-retraining".

[0044] Optionally, the sample is extracted by fine-tuning the SAM model, and the initial mask corresponding to the current target is extracted. The initial weight of the SAM backbone network is obtained, and the branch network training weight is obtained by LoRA fine-tuning; According to the initial weight and the training weight, the weight of the fine-tuned SAM model is fused. The fusion process can be realized by adding the branch network training weight and the initial weight, or by setting a proportion coefficient for the branch network training weight and the initial weight respectively, and adding the branch network training weight and the corresponding proportion coefficient, and the initial weight and the corresponding proportion coefficient. The proportion can be adjusted in actual application, and the adjustment method can be determined according to the identification result of the fine-tuned SAM model.

[0045] The sample is extracted by the fine-tuned SAM model, and the initial mask corresponding to the current target is extracted.

[0046] In actual application, the method described in the embodiment of the application can also refer to Figure 3 It is shown that in Figure 3 The pseudo label generation model is the fine-tuned SAM model, and the current target includes rice, rape and plastic greenhouse agricultural surface object targets. The above-mentioned agricultural surface object targets are only exemplary, and the specific agricultural surface object targets are not limited in actual application.

[0047] The application of the SAM fine-tuning low-altitude surface object intelligent identification sample automatic labeling method described in the embodiment of the application, the fine-tuned SAM model generated thereby, is mainly applied to automatic labeling of a large amount of high-resolution image data (such as unmanned aerial vehicle image data), application of pseudo label automatic generation technology, and is expected to greatly improve the labeling speed compared with the traditional manual labeling method. In the same period, the amount of image data and the number of labeled samples can be greatly increased, effectively alleviating the problem of sample scarcity in agricultural remote sensing application. In addition, through the setting of the model branch, accurate identification and classification of various types of crops and agricultural surface objects can be realized, and good cross-region migration ability can be achieved.

[0048] The model fine-tuning parameters obtained are stored separately from the SAM backbone network, a plug-and-play structure design is realized, the model can quickly adapt to agricultural remote sensing data in different regions and different seasons, stable recognition performance and generalization ability are maintained, and the method can be applied to more extensive agricultural application scenarios.

[0049] In addition, it should be noted that the method described in the embodiment of the application, in actual application, if a single ground object target is trained to obtain model fine-tuning parameters, and other single ground object targets are continuously trained until model fine-tuning parameters corresponding to multiple ground object targets are obtained, the model fine-tuning parameters are stored separately from the SAM backbone network; When labeling the unlabeled agricultural remote sensing image, the following technical scheme can be adopted: If the types and positions of the ground object targets contained in the agricultural remote sensing image need to be recognized and labeled, or all the ground object targets contained in the agricultural remote sensing image need to be recognized and labeled, the model fine-tuning parameters corresponding to the multiple ground object targets and the model branch are stored separately from the SAM backbone network; The unlabeled agricultural remote sensing image is input into the fine-tuning SAM model, the fine-tuning SAM model calls the model fine-tuning parameters, labels the unlabeled agricultural remote sensing image, and after completion, the next model fine-tuning parameter is called to continue labeling the agricultural remote sensing image, until all the ground object targets in the agricultural remote sensing image are labeled; In order to further improve the automatic labeling efficiency and avoid time consumption caused by labeling a small part of the area or a small part of the ground object target in the agricultural remote sensing image, the following steps can also be performed during the labeling of the agricultural remote sensing image: The area occupied by the labeled ground object target in the agricultural remote sensing image is calculated; If the area occupied by the ground object target in the agricultural remote sensing image is greater than or equal to 90% of the agricultural remote sensing image, the automatic labeling is stopped, and the labeled agricultural remote sensing image is output; Or, Select a test agricultural remote sensing image, output the labeled agricultural remote sensing image after each automatic labeling of a ground object target, and input a confirmation interface, if a technician triggers a confirmation operation in the confirmation interface, the output labeled agricultural remote sensing image is input into the fine-tuning SAM model and continues to be automatically labeled until the technician does not trigger the confirmation operation; The number of ground object targets contained in the final labeled agricultural remote sensing image is recorded; When labeling other agricultural remote sensing images of the current scene (or the same region), the selected ground object target is the ground object target contained in the final labeled agricultural remote sensing image, and the automatic labeling is completed.

[0050] Optionally, the above steps can also be: selected by the technician in the interface, and the selected ground object target is to be labeled; The unlabeled agricultural remote sensing image is input into the fine-tuning SAM model, the fine-tuning SAM model calls the model fine-tuning parameters corresponding to the selected ground object target, and labels the unlabeled agricultural remote sensing image until the selected ground object target in the agricultural remote sensing image is labeled. In actual application, if multiple ground object targets are trained to obtain model fine-tuning parameters, the model fine-tuning parameters are stored separately from the SAM main network; in the process of automatic labeling of unlabeled agricultural remote sensing images containing the above multiple ground object targets, the unlabeled agricultural remote sensing images can be input into the fine-tuning SAM model, and the fine-tuning SAM model calls the model fine-tuning parameters corresponding to the above multiple ground object targets to label the unlabeled agricultural remote sensing images.

[0051] It should be noted that the process of training multiple ground object targets to obtain model fine-tuning parameters and application is for the case where the ground object targets contained in the current scene are known, or only multiple specific ground object targets in the current scene need to be automatically labeled, and the model fine-tuning parameters can only be applied to the current scene or other scenes similar to the current scene. (The similarity described in the embodiment of the application refers to the same ground object targets contained).

[0052] Referring to Figure 4 It is also provided a low-altitude ground object intelligent identification sample automatic labeling system applying SAM fine-tuning, the system comprising a UAV data acquisition system, a parameter plug-and-play structure, and an automatic labeling model, wherein: The automatic labeling model is used to build a model LoRA fine-tuning framework based on the SAM main network, and the SAM main network and the fine-tuning model fine-tuning parameters are separated; The UAV data acquisition system is used to acquire and make samples corresponding to the current target; The automatic labeling model is used to: store the samples and classify and store the samples corresponding to different ground object targets; freeze the SAM main network, and according to the samples, incrementally train the model branch, store the obtained model fine-tuning parameters separately from the SAM main network, and classify and store the model branch parameters of different ground object targets; The parameter plug-and-play structure is used to match and adapt to multiple ground object target labeling generation; The automatic labeling model is used to: connect the trained model branch to the SAM main network, and generate an initial mask based on the complete network; The image encoder of the SAM visual model performs feature calculation on the image region covered by the initial mask to obtain relevant features; The relevant features are similarity calculated with the prototype memory bank to screen the initial mask that meets the target category features, and the automatic labeling is completed.

[0053] Optionally, the automatic labeling model is used for: From multiple ground object targets, the next target continues to be trained to obtain model fine-tuning parameters corresponding to the next target, until the training of multiple targets is completed; For different targets, the corresponding model fine-tuning parameters are matched, and the step of connecting the trained model branch to the SAM main network to extract the initial mask of the sample is executed.

[0054] Optionally, the automatic labeling model is used for: Corresponding to the self-attention calculation network layer of the SAM main network, a branch network layer is added; Corresponding to the output neural network layer of the SAM main network, a low-rank feature extraction network layer is added.

[0055] Optionally, the automatic labeling model is used for: The trained model branch is connected to the SAM main network to obtain a fine-tuned SAM model corresponding to the current target; Through the fine-tuned SAM model, the sample is extracted to extract the initial mask corresponding to the current target; The relevant features corresponding to the image in the initial mask covered area are calculated; Continue to obtain the initial mask of the next target until the initial masks and relevant features of all targets are obtained.

[0056] Optionally, the automatic labeling model is used for: The multiple features corresponding to the sample that has been accurately labeled are calculated; According to the multiple features, multiple feature vectors are obtained; According to the multiple feature vectors, a prototype memory bank is obtained.

[0057] Optionally, the automatic labeling model is used for: Set the prototype clustering algorithm; Through the prototype clustering algorithm, the similarity calculation of the relevant features and the prototype memory bank is performed, and the relevant features that meet the threshold are screened, The initial mask corresponding to the relevant features that meet the threshold is set as a pseudo label.

[0058] Optionally, the automatic labeling model is used for: The pseudo label is verified, and the sample that passes the verification is set as an effective labeled sample; The effective labeled samples are sample aggregated.

[0059] Optionally, the automatic labeling model is used for: obtaining initial weights of the SAM backbone network, and obtaining training weights of the branch network through LoRA fine-tuning; According to the initial weights and the training weights, the weights of the fine-tuned SAM model are fused. Through the fine-tuned SAM model, the sample is extracted, and an initial mask corresponding to the current target is extracted.

[0060] All the optional technical solutions described above can be combined to form optional embodiments of the present application, and will not be repeated here.

[0061] It should be noted that: the device and system provided by the above embodiments when executing the corresponding method, only the above-mentioned division of each functional module is taken as an example, and in actual application, the above-mentioned functions can be completed by different functional modules according to the needs, that is, the internal structure of the device and system is divided into different functional modules to complete all or part of the functions described above. In addition, the method, device and system embodiments provided by the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiment, which will not be repeated here.

[0062] Those skilled in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by program to instruct related hardware to complete, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be read-only memory, magnetic disk or optical disk, etc.

[0063] The above is only the preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A low-altitude ground object intelligent identification sample automatic labeling method applying SAM fine-tuning, characterized in that, The method comprises: a model LoRA fine-tuning framework based on a SAM backbone network is built, the SAM backbone network is separated from fine-tuning model fine-tuning parameters; a current target corresponding sample is collected and made by a UAV, and different ground object target samples are classified and stored; the SAM backbone network is frozen, and according to the sample, the model branch is incrementally trained, the obtained model fine-tuning parameters are stored separately from the SAM backbone network, and different ground object target model branch parameters are classified and stored; the trained model branch is connected to the SAM backbone network, and the initial mask of the sample is generated based on the complete network; the image region covered by the initial mask is calculated by the image encoder of the SAM visual model to obtain related features; the related features are calculated with the existing feature vectors in the prototype memory bank, and the initial mask of the target category feature is screened to complete automatic labeling.

2. The method of claim 1, wherein, The method further comprises: from a plurality of ground objects, the next target is continuously trained to obtain the model fine-tuning parameters corresponding to the next target, until the training of a plurality of targets is completed; for different targets, the corresponding model fine-tuning parameters are matched, and the steps of connecting the trained model branch to the SAM main network and extracting the initial mask of the sample are executed.

3. The method of claim 1, wherein, The model LoRA fine-tuning framework based on the SAM backbone network comprises: a self-attention calculation network layer corresponding to the SAM backbone network, and a branch network layer is added; an output neural network layer corresponding to the SAM backbone network, and a low-rank feature extraction network layer is added.

4. The method of claim 3, wherein, The method further comprises: connecting the trained model branch to the SAM backbone network to obtain a fine-tuning SAM model corresponding to the current target, i.e., a complete network; extracting the initial mask corresponding to the current target from the sample through the fine-tuning SAM model; calculating the related features corresponding to the image region covered by the initial mask; continuing to obtain the initial mask of the next target until the initial masks and related features of all targets are obtained.

5. The method of claim 4, wherein, Before the related features are calculated with the existing feature vectors in the prototype memory bank to complete automatic labeling, the method further comprises: calculating a plurality of features corresponding to the samples that have been accurately labeled; obtaining a plurality of feature vectors corresponding to the plurality of features; obtaining the prototype memory bank according to the plurality of feature vectors.

6. The method of claim 5, wherein, The method further comprises: setting a prototype clustering algorithm; calculating the similarity between the related features and the prototype memory bank through the prototype clustering algorithm, and screening the related features with a similarity satisfying a threshold value; setting the initial mask corresponding to the related features with a similarity satisfying a threshold value as a pseudo label.

7. The method of claim 6, wherein, The method further comprises: Verify the pseudo label, and set the sample with successful verification as an effective labeled sample; Sample collection is performed on the effective labeled sample.

8. The method of claim 7, wherein, The branch network is connected to the SAM main network, and the initial mask generation is performed on the sample based on the complete network, and the method further comprises: Obtaining the initial weight of the SAM main network, and obtaining the training weight of the branch network through LoRA fine-tuning; According to the initial weight and the training weight, the weight of the fine-tuned SAM model is obtained; Through the fine-tuned SAM model, the initial mask corresponding to the current target is extracted.

9. A low-altitude ground object intelligent identification sample automatic labeling system applying SAM fine-tuning, characterized in that, The system comprises a UAV data acquisition system, a parameter plug-and-play structure and an automatic labeling model, wherein: The automatic labeling model is used to build a model LoRA fine-tuning framework based on a SAM main network, and the SAM main network and the fine-tuning model are separated from the fine-tuning parameters; The UAV data acquisition system is used to acquire and produce samples corresponding to the current target; The automatic labeling model is used to: Store the sample, and store the sample corresponding to different ground object targets in different categories Freeze the SAM main network, and perform incremental training on the model branch according to the sample, store the obtained model fine-tuning parameters separately from the SAM main network, and store the model branch parameters of different ground object targets in different categories; The parameter plug-and-play structure is used to match and adapt to multi-class ground object labeling generation; The automatic labeling model is used to: Connect the trained model branch to the SAM main network, and generate the initial mask based on the complete network; Through the image encoder of the SAM visual model, the feature calculation of the image region covered by the initial mask is performed, and the related features are obtained; The related features are calculated with the prototype memory bank, and the initial mask meeting the target category features is screened to complete the automatic labeling.

10. The system of claim 9, wherein, The automatic labeling model: Corresponding to the self-attention calculation network layer of the SAM main network, a branch network layer is added; Corresponding to the output neural network layer of the SAM main network, a low-rank feature extraction network layer is added.

Images