Processing method and system for full-scene ground object segmentation based on visual large model

Abstract

The invention discloses a processing method and system for full-scene ground object segmentation based on a visual large model, and belongs to the technical field of computer vision. Comprising the following steps: acquiring video data shot by an unmanned aerial vehicle, and preprocessing image data in a video; the method comprises the following steps: detecting categories in image data by using Grounding DINO through preset information, and obtaining an instance segmentation mask of a detected frame by using an SAM segmentation model; using CLIPSeg to obtain a primary segmentation mask of the category; determining a sampling point in the primary segmentation mask, and inputting the sampling point into the SAM segmentation model to obtain a secondary segmentation mask of the category; and combining the primary segmentation mask with the secondary segmentation mask to obtain a panoramic segmentation result. Compared with the prior art, the method has the beneficial effects that the problem that a traditional SAM segmentation model does not have the ability of text recognition and semantic analysis between objects is solved, and full-scene segmentation of the view angle picture of the unmanned aerial vehicle according to the cue word is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically, to a processing method and system for full-scene object segmentation based on a large visual model. Background Art

[0002] With the rapid development of drone technology, its application scenarios are becoming increasingly complex and diverse. With its high-altitude perspective and wide coverage, drones have shown great potential in monitoring, reconnaissance, agriculture, surveying and mapping, etc. However, accurately acquiring and analyzing image data from the perspective of drones has become a key issue in the current technological development.

[0003] Traditional image segmentation methods generally face two major challenges when dealing with drone images: slow processing speed and insufficient accuracy. The images taken by drones are often of extremely high resolution, with complex and changeable scenes and rich details in the images, which greatly increases the difficulty of image processing. High-resolution images not only mean a sharp increase in the amount of data, but also mean that more sophisticated processing algorithms are needed to capture and identify key information in the image. In addition, the environmental conditions in which drones are located during flight are complex and changeable, such as changes in light intensity, cloudy, rainy and snowy weather conditions, etc. These environmental factors will have a significant impact on image quality. Light changes may cause uneven brightness of the image, and weather conditions may introduce noise and blur, all of which further increase the difficulty of image analysis.

[0004] In the field of computer vision, image semantic segmentation and instance segmentation are two important image analysis methods. The task of image semantic segmentation is to predict the semantic category of each pixel; the task of instance segmentation is to predict the pixel area contained in each instance object. Semantic segmentation aims to predict the semantic category of each pixel in the image, while instance segmentation goes a step further and aims to identify and segment each instance object in the image. However, these two methods also face the problems of slow processing speed and insufficient accuracy when processing drone images. In order to overcome these challenges, panoramic segmentation technology came into being. Panoramic segmentation was jointly proposed by FAIR and Heidelberg University in Germany. It combines the advantages of semantic segmentation and instance segmentation, aiming to assign category labels and instance IDs to each pixel, thereby generating a global and unified segmented image. However, the existing panoramic segmentation technology still needs to be further improved in processing high-resolution images, capturing complex details, and quickly adapting to dynamically changing environmental conditions.

[0005] In the related technology, for example, Chinese patent CN113538441A provides a method for processing an image segmentation model, an image processing method and a device. It includes: obtaining a sample image of the target scene; performing image segmentation on the sample image through a lightweight segmentation model to be trained, and obtaining a first segmentation result in the sample image in the lightweight segmentation model to be trained; performing image segmentation on the sample image through a trained teacher segmentation model, and obtaining a second segmentation result in the sample image in the trained teacher segmentation model; constructing a knowledge distillation loss function based on the first segmentation result and the second segmentation result; when the sample image is an unlabeled sample image, the model parameters of the lightweight segmentation model are updated according to the knowledge distillation loss function, and then training is continued until a lightweight segmentation model for the target scene is obtained. The shortcoming of this solution is that it does not take into account the current problems faced by the field of drone image analysis, such as complex and changeable environmental conditions and the inability to identify the categories of segmented objects.

[0006] From the above, we can see that the relevant technology does not provide any technical inspiration on how to identify and segment object categories in drone image analysis. Summary of the invention

[0007] 1. Technical issues to be solved

[0008] In view of the problem of how to identify and segment object categories in drone image analysis in the prior art, the present invention provides a processing method and system for full-scene ground object segmentation based on a large visual model, which can obtain the rectangular frame and mask information generated by the Grounding DINO model and the CLIPSeg model, and realize panoramic segmentation.

[0009] 2. Technical solution

[0010] The purpose of the present invention is achieved through the following technical solutions.

[0011] The content of this application is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this application is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0012] Some embodiments of the present application propose a processing method and system for full-scene object segmentation based on a large visual model to solve the technical problems mentioned in the above background technology section.

[0013] As the first aspect of the present application, some embodiments of the present application provide a processing method for full-scene ground object segmentation based on a large visual model, including the following steps: obtaining video data taken by a drone, and preprocessing the image data in the video; using Grounding DINO to detect categories in the image data through preset information, and using the SAM segmentation model to obtain the instance segmentation mask of the detected box; using CLIPSeg to obtain a primary segmentation mask of the category; determining sampling points in the primary segmentation mask, and inputting the sampling points into the SAM segmentation model to obtain a secondary segmentation mask of the category; combining the primary segmentation mask with the secondary segmentation mask to obtain a panoramic segmentation result.

[0014] Furthermore, the video data includes current frame image data of the video shot by the drone, or a set of key frame image data obtained according to a preset interval time.

[0015] Furthermore, the preprocessing includes image denoising and image correction; the image correction includes geometric correction and radiation correction.

[0016] Furthermore, geometric correction corrects the geometric characteristics of the image, including the geometric position, shape, size and orientation of the image; radiation correction corrects the radiation characteristics of the image, including the brightness, contrast and color of the image.

[0017] Furthermore, the preset information includes object name and location information.

[0018] Furthermore, by inputting preset information into Grounding DINO, a target rectangular frame providing target positioning information is generated.

[0019] Furthermore, the SAM segmentation model performs secondary segmentation of the image category based on the target rectangular box.

[0020] As the second aspect of the present application, some embodiments of the present application provide a system for a processing method for full-scene ground object segmentation based on a large visual model, including an image preprocessing module that obtains video data taken by a drone and preprocesses the image data in the video; a detection module: uses Grounding DINO to detect categories in image data through preset information, and uses the SAM segmentation model to obtain a segmentation mask; a segmentation module: uses CLIPSeg to obtain a primary segmentation mask of the category; determines sampling points in the primary segmentation mask, and inputs the sampling points into the SAM segmentation model to obtain a secondary segmentation mask of the category; combines the primary segmentation mask with the secondary segmentation mask to obtain a panoramic segmentation result.

[0021] As the third aspect of the present application, some embodiments of the present application provide a processing device for full-scene object segmentation based on a large visual model, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, the processing method for full-scene object segmentation based on a large visual model is implemented.

[0022] As the fourth aspect of the present application, some embodiments of the present application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-mentioned processing method for full-scene object segmentation based on a large visual model.

[0023] 3. Beneficial effects

[0024] Compared with the prior art, the advantages of the present invention are: it overcomes the problem that the original SAM segmentation model does not have the ability of text recognition and semantic analysis, and cannot identify the category of the segmented object. The prompt is input into the clipseg multimodal model, and the binary classification region segmentation of the image is realized through the text prompt to generate a mask. Image segmentation is generated according to any prompt during the test. The problem that the SAM large model does not have the ability of text recognition and semantic analysis between objects is solved, and the full scene segmentation of the drone perspective image is realized according to the prompt words. The SAM model has the ability of text recognition and identifies the category of the segmented object; finally, the SAM model has the ability of semantic analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flowchart of a processing method for full-scene object segmentation based on a large visual model in one embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a target frame input into a Dino detection model to obtain an output result of a target detection category in an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of a result after SAM segmentation is obtained by inputting a target frame obtained after processing into a SAM model as a prompt in one embodiment of the present invention;

[0028] Figure 4 A schematic diagram of sampling points in a segmentation mask for each “ground feature” category in one embodiment of the present invention;

[0029] Figure 5 Schematic diagram of the segmentation effect of the SAM segmentation model in one embodiment of the present invention;

[0030] Figure 6 FIG. 4 is a schematic diagram of the segmentation effect of another SAM segmentation model in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Combination Figures 1 to 6 The processing method for full scene ground object segmentation based on the visual large model of the present invention obtains images through the camera of the unmanned aerial vehicle, and performs a series of preprocessing and segmentation processing on the image data to achieve high-precision full scene ground object segmentation. The processing method for full scene ground object segmentation based on the visual large model includes the following steps:

[0033] S1. Image preprocessing

[0034] Obtain video data taken by drones and perform image preprocessing on the video data to improve image quality and provide clear and accurate input data for subsequent segmentation processing.

[0035] Specifically, obtaining video data shot by a drone refers to obtaining current frame image data of the video shot by the drone or a set of key frame image data obtained according to a preset interval time.

[0036] The key frame image data set obtained at a preset interval can reduce the amount of data and processing time. By presetting a time interval, key frames are extracted from the video. The selection of key frames is based on factors such as changes in video content, frame rate, and timestamp.

[0037] In a specific embodiment, the image preprocessing includes image denoising and image correction. The image preprocessing process is as follows:

[0038] S101, Image Denoising:

[0039] When a drone performs a shooting task, the image data collection process is often interfered by sensor noise and environmental noise, which will undoubtedly reduce the overall quality of the image, and then affect the accuracy and efficiency of subsequent image segmentation processing, resulting in a decrease in segmentation accuracy. Therefore, image denoising technology is first used to eliminate these noises.

[0040] Image denoising methods include spatial domain filters and frequency domain filters. Among them, spatial domain filters include mean filters and median filters. Use mean filters and median filters to smooth images to reduce noise. The mean filter replaces the central pixel value by calculating the average value of the surrounding pixels, that is, calculates the average value of the pixels in the neighborhood around each pixel in the image, and uses this average value as the new value of the pixel, thereby achieving the purpose of smoothing the image and reducing noise. The median filter replaces the central pixel value by taking the median value of the surrounding pixels. This method has a significant effect on removing point noise such as salt and pepper noise, and can better preserve the edge and detail information of the image.

[0041] On the other hand, the frequency domain filtering method converts the image from the spatial domain to the frequency domain and reduces the noise by processing the spectrum of the image. In this process, high-pass filters and low-pass filters play an important role. The high-pass filter allows high-frequency signals to pass through, which is used to remove the blur effect in the image and enhance the details and edge information of the image; while the low-pass filter allows low-frequency signals to pass through, which is used to remove high-frequency noise in the image, such as particle noise and texture noise.

[0042] After image denoising, the noise in the image can be significantly reduced, making the image clearer, which provides more accurate input data for subsequent image segmentation processing. Clear images help the segmentation algorithm to more accurately identify the boundaries of objects, thereby improving segmentation accuracy. At the same time, the denoised image requires less noise information to be processed in subsequent processing, reducing computational complexity and improving processing efficiency.

[0043] S102, Image Correction:

[0044] When drones perform shooting tasks, the image data obtained is often affected by various geometric distortions, including lens distortion and perspective distortion. These distortions will affect the true shape and position of objects in the image, and thus affect subsequent image analysis and applications. Therefore, in this solution, image correction is used to correct these geometric distortions and restore the true appearance of the image.

[0045] Image correction refers to the process of eliminating or correcting the geometric errors of remote sensing images. There is often geometric distortion in the videos shot by drones, especially in the edge areas of the lens, where the images often show obvious nonlinear deformations, which seriously interfere with the accurate understanding of the image content. Therefore, image correction is used to correct these geometric distortions to ensure that the objects in the image on the two-dimensional plane can accurately reflect their true shape and position in three-dimensional space.

[0046] Image correction includes geometric correction and radiation correction. Geometric correction is to correct the geometric characteristics of the image, and ensure that the size, direction and position of the objects in the image are consistent with the actual situation. Radiation correction is to adjust the radiation characteristics of the image. Radiation correction can further improve the image quality.

[0047] Specifically, the geometric characteristics of an image include the geometric position, shape, size, and orientation of the image, etc. The radiometric characteristics of an image include the brightness, contrast, and color of the image, etc.

[0048] In a specific embodiment, the process of geometric correction is as follows:

[0049] Preprocessing:

[0050] Adjusting the brightness and contrast of the image and ensuring the stability of the image in the lighting environment can significantly reduce image quality problems caused by uneven lighting or improper exposure, and provide a clear and uniform image foundation for subsequent correction.

[0051] Calibration Instrument:

[0052] Use a correction instrument to perform preliminary correction on the image to ensure that the image scale is correct, thereby preliminarily eliminating image deformation problems caused by shooting angle or lens distortion.

[0053] Specifically, the correction instrument can be an optical mechanical corrector or an electronic corrector. Their main function is to ensure that the image is in the correct scale, that is, the size of the object in the image is consistent with the actual size.

[0054] Mathematical model application:

[0055] Based on the preliminary correction, mathematical models and algorithms are further applied to perform geometric transformation on the image. These models include but are not limited to affine transformation and projection transformation. These models can be adaptively adjusted according to the geometric features of the image, so as to accurately correct the geometric distortion in the image and ensure that the corrected image is consistent with the real scene. Through this step, the shape, direction and position of the objects in the image will be significantly improved, making it closer to the real-world scene.

[0056] Resampling Techniques:

[0057] The pixel values ​​of the corrected image are interpolated using resampling technology to ensure the continuity and smoothness of the image after geometric correction, avoiding problems such as pixel breakage or distortion. Through this step, the corrected image will present a more delicate and natural visual effect.

[0058] Specifically, resampling techniques include nearest neighbor interpolation, bilinear interpolation, and cubic convolution interpolation. Their main functions are

[0059] In a specific embodiment, the radiation correction process is as follows:

[0060] Screen calibration:

[0061] Screen calibration is performed by adjusting the brightness and contrast of the display device to eliminate image distortion caused by device differences or improper settings, ensuring that the image is stable and clear when displayed, laying a good foundation for subsequent radiation calibration.

[0062] Opto-mechanical correction:

[0063] Optical mechanical correction is the use of optical mechanical principles to physically correct images.

[0064] Optical mechanical correction involves adjusting the focal length of the lens to optimize image clarity and using physical tools such as correction plates to eliminate lens distortion. This step can significantly reduce image quality issues caused by imperfections in the optical system.

[0065] Combination of electronic computers and optical instruments:

[0066] Joint calibration is performed using electronic computers and optical instruments.

[0067] Specifically, electronic computers are combined with optical instruments to perform more precise corrections. By calculating the geometric distortion parameters in the image and applying the corresponding correction algorithm, the distortion in the image can be accurately corrected. This step fully utilizes the advantages of modern computing technology to achieve high-precision correction of the image.

[0068] Geocoding and resampling:

[0069] Based on the geometric correction of the image, geocoding and resampling are further performed. Geocoding is to combine the corrected image with the geographic coordinate system to ensure that the objects in the image can be accurately located in the geographic space. The resampling technology is to convert the image into the required map projection format while maintaining the continuity and smoothness of the image. Through this step, the corrected image not only has accurate geometric features, but also has geographic coordinate information consistent with the real scene.

[0070] Through the above radiation correction steps, a preprocessed image is finally obtained. The geometric distortion in the preprocessed image can be effectively eliminated, so that the objects in the image can accurately reflect their true shape and position on the two-dimensional plane. The corrected image is clearer and more accurate, providing a solid foundation for subsequent image segmentation processing and object recognition. In addition, the corrected image is consistent with the real scene, providing reliable data support for panoramic segmentation processing.

[0071] S2. Multi-target detection

[0072] Based on the preset information, Grounding DINO is used to detect the categories in the image data, and SAM is used to obtain the instance segmentation mask of the detected box. The image data refers to the preprocessed image data in S1.

[0073] Specifically, urban target detection based on Grounding DINO is combined with SAM segmentation model for segmentation, multi-target detection is achieved through preset information, and target rectangular boxes are generated.

[0074] This solution adopts urban target detection technology based on Grounding DINO and combines it with the SAM segmentation model to achieve multi-target detection and segmentation.

[0075] Grounding DINO is an open collection of object detectors. By combining the Transformer-based detector DINO and the language-based pre-trained model, it can detect any manually entered category name or related expression target. The Grounding DINO system not only inherits the excellent detection performance of DINO, but also realizes the detection capability of any manually entered category name or related expression target by introducing the semantic understanding ability of the language model. Users only need to use simple preset information to guide the Grounding DINO visual large model to accurately locate and detect multiple targets in the image, and generate accurate target rectangles.

[0076] By inputting preset information into the Grounding DINO visual model, the model can detect multiple targets based on text prompts and generate target rectangular boxes. These target rectangular boxes provide accurate target positioning information for the subsequent accurate segmentation of the SAM segmentation model.

[0077] Specifically, the preset information is a custom text prompt. The preset information is an instruction or description provided to the SAM segmentation model according to the obtained requirements, and the preset information includes the object name and location information.

[0078] The SAM segmentation model uses this target location information to perform fine segmentation of the target in the image. Because the SAM model is "promptable", it can adapt to new data sets and tasks without first labeling and training the data. This means that when faced with a new segmentation task, the SAM segmentation model can accurately segment the objects in the image with just a simple prompt word information.

[0079] Detect specified targets based on text descriptions using Grounding DINO. Input custom text prompts into the Grounding DINO visual model, use text prompts to detect multiple targets, and generate target rectangular boxes. Specifically, when users input text describing the target, the Grounding DINO system can respond quickly, detect and locate multiple targets in the image based on the text prompts, and generate rectangular boxes that closely fit the target contours. Subsequently, the SAM segmentation model uses the positioning information provided by these rectangular boxes to perform more detailed segmentation processing on the targets in the image, ultimately generating clear and accurate segmentation results.

[0080] In a specific embodiment, a city object detection category is input, for example: "cityobject_prompts": ['car', 'bus', 'person'], and is input into the Grounding DINO detection model to obtain an output result of a target box of the target detection category, and the obtained target box is input into the SAM model as a prompt to obtain the result after SAM segmentation.

[0081] Specifically, the specific steps of multi-target detection are as follows:

[0082] (1) Input the image and the city object detection category to be detected "cityobject_prompts": ['car', 'bus', 'person'];

[0083] (2) Input into the Grounding DINO detection model to obtain the output result as the target box of the target detection category;

[0084] (3) Input the target box obtained after the above processing into the SAM model as a prompt to obtain the result after SAM segmentation.

[0085] like Figure 2 As shown in FIG. 1 , a schematic diagram of a target box whose output result is a target detection category is input into the Grounding DINO detection model.

[0086] like Figure 3 As shown, the target box obtained after processing is input into the SAM segmentation model as a prompt to obtain the result image after SAM segmentation.

[0087] Grounding DINO can detect any manually entered category name or related expression target, greatly improving the flexibility of target detection. The "promptable" nature of the SAM segmentation model enables it to adapt to new data sets and tasks with zero samples, without the need for data annotation and training, reducing time and cost. The "promptable" nature of the SAM segmentation model means that it can accept and process various forms of prompts to achieve segmentation of different targets in the image. This property makes the SAM model extremely flexible and adaptable, and can cope with various complex scenarios and new tasks.

[0088] By combining DINO's target positioning information and the fine segmentation capabilities of the SAM segmentation model, accurate segmentation of targets in images can be achieved. This not only greatly improves the accuracy and efficiency of target detection and segmentation, but also improves flexibility, openness and adaptability. No matter what complex scene or new task you are facing, you can quickly and accurately detect and segment targets in images with simple prompt word information.

[0089] S3. Binary region segmentation of images

[0090] Use CLIPSeg to obtain the first-class segmentation mask; determine the sampling points in the first-class segmentation mask, and input the sampling points into the SAM segmentation model to obtain the second-class segmentation mask; combine the first-class segmentation mask with the second-class segmentation mask to obtain the panoramic segmentation result.

[0091] Specifically, the first segmentation mask is the foreground mask of the initial segmentation, and the second segmentation mask is the background mask of the further segmentation obtained after segmentation by the SAM segmentation model; finally, the panoramic segmentation result is obtained, that is, the panoramic segmentation label obtained by combining the first segmentation mask with the second segmentation mask.

[0092] In a specific embodiment, CLIP-based object classification technology is used in combination with the SAM segmentation model to achieve binary classification region segmentation of the image. Binary classification region segmentation of the image is achieved through text prompts, sampling points in the coarse segmentation mask are used as SAM prompts to generate a fine segmentation mask, and finally a mask result combining urban target and object classification is obtained.

[0093] CLIP (Contrastive Language-Image Pre-training), as a multimodal vision and text learning method, provides strong support for image classification and segmentation tasks by virtue of its rich visual and text features learned from massive image-text pairs.

[0094] The custom text prompt is input into the clipseg multimodal model, which implements binary region segmentation of the image based on the text prompt and generates a mask.

[0095] Specifically, in order to achieve binary region segmentation of images, the clipseg multimodal model is introduced. This model can receive user-defined text prompts and perform accurate binary region segmentation of images based on these prompts, while generating high-quality masks.

[0096] More specifically, the prompt can be a descriptive text or an image with specific meaning, thus providing a unified and flexible solution for three common segmentation tasks: foreground segmentation, background segmentation, and specific category segmentation, creating a unified model.

[0097] In order to further enhance the segmentation capability of the ClipSeg multimodal model, the CLIP model is extended to support image or prompt-based segmentation queries.

[0098] Specifically, a Transformer-based architecture is used to extend the CLIP model so that it can handle complex segmentation tasks. During the testing phase, users can enter any prompt, and the clipseg multimodal model can respond quickly and generate accurate image segmentation results.

[0099] At the same time, combining the SAM segmentation model with the clipseg multimodal model can further improve the accuracy and robustness of segmentation. With its powerful feature extraction and fine segmentation capabilities, the SAM segmentation model can make full use of the rich feature information provided by the CLIP model to perform more accurate and detailed segmentation of targets in the image. This combination not only improves the accuracy and robustness of segmentation, but also further enhances the adaptability and flexibility of the model.

[0100] In a specific embodiment, the process of binary classification region segmentation of an image is as follows: input the feature segmentation category, for example: stuff_category_names = ["building", "road", "sky", "trees", "sidewalk"], and input it into the clipseg multimodal model to obtain the binary classification region segmentation of the image.

[0101] like Figure 4 Shown, for each “ground object” category, the sampling points in the coarse segmentation mask.

[0102] By integrating CLIP and SAM models, this solution realizes multimodal segmentation based on text and image prompts. This solution not only improves the flexibility and accuracy of segmentation, but also provides new ideas and methods for research and application in the fields of image processing and computer vision. The clipseg multimodal model can provide efficient, accurate and reliable solutions for segmentation tasks such as foreground segmentation, background segmentation and specific category segmentation.

[0103] By combining CLIP's object classification technology, clipseg multimodal model and SAM segmentation model, this solution can successfully achieve the binary classification region segmentation task of the image. The realization of flexible, accurate and efficient multi-target detection and image binary classification region segmentation also provides new ideas and methods for research and application in the field of image processing and computer vision.

[0104] like Figures 5 to 6 As shown in the figure, the image category is automatically segmented by custom text prompts. It can be seen that the SAM segmentation model can segment objects in the image at the pixel level, but the SAM model does not have the ability to recognize text and semantic analysis, and cannot identify the category of the segmented object, so it is not practical in many applications; Figure 5 and Figure 6 As shown, (A) is a picture taken by a drone, (B) is a semantic map after the model is segmented by categories, (C) is the effect map after the segmented image is superimposed with the original image, that is, the effect map after the superposition of (A) and (B), (D) shows the statistics of the pixel values ​​after segmentation, that is, the statistics of the category pixel values ​​in (B), and different colors represent different category effect maps.

[0105] The processing method of the full-scene object segmentation based on the visual large model of the present invention, by inputting a specific prompt into the innovative clipseg multimodal model, the model can realize the accurate segmentation of the binary classification area of ​​the image according to the text prompt and automatically generate the corresponding mask. This feature enables the model to flexibly perform image segmentation according to any prompt input by the user during the testing phase. It overcomes the limitation of the traditional SAM segmentation model that lacks text recognition and semantic analysis capabilities, and thus cannot accurately identify and segment the categories of objects in the image.

[0106] Specifically, the processing method of full-scene ground object segmentation based on the visual large model of the present invention breaks through the limitations of the SAM large model in text recognition and semantic understanding between objects, and realizes the full-scene segmentation of drone-view images based on prompt words. This not only gives the SAM model the ability to recognize text, enabling it to accurately identify the category of the segmented object, but also greatly enhances its semantic analysis ability, and finally successfully upgrades the SAM model to an advanced segmentation model with dual capabilities of text recognition and semantic analysis.

[0107] Combination Figures 1 to 6 , a system for processing a method for segmenting full-scene objects based on a large visual model, comprising:

[0108] The image preprocessing module obtains the video data taken by the drone and preprocesses the image data in the video;

[0109] Detection module: Based on the preset information, use Grounding DINO to detect the categories in the image data and use the SAM segmentation model to obtain the segmentation mask;

[0110] Segmentation module: Use CLIPSeg to obtain the first segmentation mask of the category; determine the sampling points in the first segmentation mask, and input the sampling points into the SAM segmentation model to obtain the second segmentation mask of the category; combine the first segmentation mask with the second segmentation mask to obtain the panoramic segmentation result.

[0111] The processing method and system of the full-scene ground object segmentation based on the visual large model of the present invention solves the problem that the original SAM segmentation model does not have the ability of text recognition and semantic analysis and cannot identify the category of the segmented object by combining the SAM-DINO-CLIP visual model. First, the image taken by the drone is preprocessed, and at the same time, in order to enable the SAM (segment anything model) large model to have the ability of text recognition, Grounding DINO is added to detect the specified target according to the text description to obtain the object frame as the prompt of SAM for segmentation to obtain the text recognition ability. In order to enable SAM to have the ability of semantic analysis, further, the prompt is input into the clipseg multimodal model, and the binary classification area segmentation of the image is realized through the text prompt to generate a mask. Image segmentation is generated according to any prompt during the test. This solves the problem that the SAM large model does not have the ability of text recognition and semantic analysis between objects, and realizes the full-scene segmentation of the drone perspective picture according to the prompt word. It effectively solves the problems of slow processing speed, insufficient precision, complex and changeable environmental conditions, and inability to identify the category of the segmented object in the current field of drone image analysis.

[0112] The processing device designed for the processing method of full-scene object segmentation based on a large visual model can be a hardware device, including one or more processors and a memory, the memory stores a program, and is configured to execute the following steps by one or more processors:

[0113] Obtain video data taken by drones and pre-process image data in the video;

[0114] Based on the preset information, use Grounding DINO to detect categories in image data, and use the SAM segmentation model to obtain instance segmentation masks of detected boxes;

[0115] Use CLIPSeg to obtain the first segmentation mask of the category; determine the sampling points in the first segmentation mask, and input the sampling points into the SAM segmentation model to obtain the second segmentation mask of the category; combine the first segmentation mask with the second segmentation mask to obtain the panoramic segmentation result.

[0116] The storage medium designed for the processing method of full-scene feature segmentation based on a large visual model can be a computer-readable storage medium, which stores a computer program. When the computer program is executed by the processor, the method for constructing the above-mentioned three-dimensional high-precision forest belt denoising prediction model is implemented.

[0117] How to realize the device can be realized in the existing technology. I will not elaborate on it in detail again. The following explains the possibility of corresponding realization in principle. In the 1990s, the improvement of a technology could be clearly distinguished as hardware improvement (for example, improvement of circuit structure such as diode, transistor, switch) or software improvement (improvement of method flow). However, with the development of technology, many improvements of method flow today can be regarded as direct improvements of hardware circuit structure.

[0118] Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be realized by hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit, and its logic function is determined by the user's programming of the device. Designers program by themselves to "integrate" a digital system on a PLD, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when writing program development, and the original code before compilation must also be written in a specific programming language, which is called Hardware Description Language (HDL), and there is not only one HDL, but many kinds, such as Verilog. Those skilled in the art should also be aware that it is easy to obtain a hardware circuit that implements the logical method flow by simply performing some logic programming on the method flow using the above-mentioned hardware description languages ​​and programming it into an integrated circuit.

[0119] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a computer-readable medium such as a microprocessor or processor and a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include but are not limited to the following microcontrollers: ATMEL AT89S52, microchip pic16c57. The memory controller can also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0120] The systems, devices, modules or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices. For the convenience of description, the above devices are described separately by function into various units. Of course, when implementing this specification, the functions of each unit can be implemented in the same or more software and / or hardware.

[0121] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in the blocks or multiple blocks. In a typical configuration, the computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0123] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0124] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0125] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0127] The above schematically describes the invention and its implementation methods, which is not restrictive. Without departing from the spirit or basic features of the invention, the invention can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention. The actual structure is not limited to this, and any figure mark in the claims should not limit the claims involved. Therefore, if a person of ordinary skill in the art is inspired by it, without departing from the purpose of the invention, a structural method and an embodiment similar to the technical solution are designed without creativity, which should all fall within the scope of protection of this patent. In addition, the word "including" does not exclude other elements or steps, and the word "one" before the element does not exclude the inclusion of "multiple" elements. The multiple elements stated in the product claim can also be implemented by one element through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

Claims

1. A method for processing full scene object segmentation based on a large visual model includes the following steps: Obtain video data taken by drones and pre-process image data in the video; Based on the preset information, use Grounding DINO to detect categories in image data, and use the SAM segmentation model to obtain instance segmentation masks of detected boxes; Use CLIPSeg to obtain the first segmentation mask of the category; determine the sampling points in the first segmentation mask, and input the sampling points into the SAM segmentation model to obtain the second segmentation mask of the category; combine the first segmentation mask with the second segmentation mask to obtain the panoramic segmentation result.

2. The method for processing full scene object segmentation based on a large visual model according to claim 1, characterized in that: The video data includes the current frame image data of the video shot by the drone, or a set of key frame image data obtained according to a preset interval time.

3. The method for processing full scene ground object segmentation based on a large visual model according to claim 1, characterized in that: Preprocessing includes image denoising and image correction; image correction includes geometric correction and radiation correction.

4. The method for processing full scene ground object segmentation based on a large visual model according to claim 3 is characterized in that: Geometric correction is performed based on the geometric characteristics of the image, including the geometric position, shape, size and orientation of the image; Radiometric calibration is used to calibrate the radiometric characteristics of an image, including its brightness, contrast, and color.

5. The method for processing full scene ground object segmentation based on a large visual model according to claim 1, characterized in that: The preset information includes the object name and location information.

6. The method for processing full scene ground object segmentation based on a large visual model according to claim 5, characterized in that: By inputting the preset information into Grounding DINO, a target rectangular box providing target positioning information is generated.

7. The method for processing full scene ground object segmentation based on a large visual model according to claim 6, characterized in that: The SAM segmentation model performs secondary segmentation of the image category based on the target rectangular box.

8. A system based on the method for processing full scene object segmentation based on a large visual model according to any one of claims 1 to 7, characterized in that: It includes an image preprocessing module to obtain video data taken by the drone and preprocess the image data in the video; Detection module: Based on the preset information, use Grounding DINO to detect the categories in the image data and use the SAM segmentation model to obtain the segmentation mask; Segmentation module: Use CLIPSeg to obtain the first segmentation mask of the category; determine the sampling points in the first segmentation mask, and input the sampling points into the SAM segmentation model to obtain the second segmentation mask of the category; combine the first segmentation mask with the second segmentation mask to obtain the panoramic segmentation result.

9. A processing device for full scene object segmentation based on a large visual model, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the processing method for full-scene object segmentation based on a large visual model as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the processing method for full-scene object segmentation based on a large visual model as described in any one of claims 1 to 7.

Images