Small target object detection method and system based on urban multi-mode large model

Abstract

The invention provides a small target object detection method and system based on an urban multi-modal large model, and relates to the technical field of smart cities, and the method comprises the following steps: S1, obtaining an original picture with a small target object detection frame and an instruction text; s2, compressing the original picture with the small target object detection frame to obtain a simple picture; s3, cutting or compressing the original picture with the small target object detection frame to obtain a local environment map; s4, performing traversal cutting on the original picture with the small target object detection frame by taking the left upper corner of the small target object detection frame as a starting point to obtain a plurality of local internal pictures; s5, inputting the brief graph, the local environment graph and the plurality of local internal graphs into the urban multi-modal large model to obtain visual embedding features with graph interval marks; and S6, obtaining a prediction text according to the visual embedding features with the graph interval marks and the instruction text embedding features.

Description

Technical Field

[0001] The present invention relates to the technical field related to smart cities, and in particular to a small target object detection method and system based on a large multimodal urban model. Background Art

[0002] Currently, general-purpose multimodal large models have excellent whole-image understanding capabilities, significantly improving object recognition accuracy. However, in urban management, small object detection tasks are often encountered, such as scattered garbage, scattered waste, and small advertisements. These detected objects are relatively small relative to the original image.

[0003] For small target detection tasks, the general practice of general multimodal large models is to first compress the original image to the input scale (448x448), then traverse the entire original image, divide the original image into multiple local segmented images, and finally input the original image and multiple local segmented images into the multimodal large model in sequence.

[0004] This practice of equally partitioning the original image and processing all the local slices reduces the efficiency of large multimodal models. In particular, for today's ultra-high-resolution (4K) urban management snapshots, excessive local slices can cause the tokens of the local images to overwhelm the tokens of the overall image, or even exceed the context length of the large model. Summary of the Invention

[0005] The purpose of the present invention is to provide a small target object detection method and system based on an urban multimodal large model. When a small target object detection frame suspected to be a small target object appears in the original image, the original image is processed accordingly according to the input scale of the urban multimodal large model. The simplified image, local environment image and local internal image obtained after processing are sequentially spliced ​​through image interval marks to form a visual embedding feature with image interval marks. The visual embedding feature can improve the efficiency of the multimodal large model in detecting small targets.

[0006] In order to solve the above technical problems, the present invention adopts the following solutions:

[0007] A small target object detection method based on a large multimodal urban model includes the following steps:

[0008] S1. Obtain an original image with a small target object detection frame and the corresponding instruction text, and input the original image with the small target object detection frame into the urban multimodal large model;

[0009] S2. Compress the original image with the small target object detection frame according to the input scale of the urban multimodal large model to obtain the corresponding simplified image;

[0010] S3. Cropping or compressing the original image with the small target object detection frame according to the input scale of the urban multimodal large model and the input scale of the small target object detection frame to obtain a corresponding local environment map;

[0011] S4, performing traversal cutting on the original image with the small target object detection frame starting from the upper left corner of the small target object detection frame to obtain several corresponding local internal images;

[0012] S5. Inputting the simplified image, the local environment image, and several local internal images into the urban multimodal large model, and obtaining visual embedding features with image interval marks through the urban multimodal large model;

[0013] S6. Obtain instruction text embedding features obtained after processing the instruction text, and obtain predicted text embedding features based on the visual embedding features with image interval marks and the instruction text embedding features.

[0014] A further preferred technical solution is: the original picture with the small target object detection frame refers to the collected city street scene picture input into the target recognition model for target recognition, and the small target object detection frame displayed on the original picture is obtained through target recognition.

[0015] A further preferred technical solution is that the scales of the simplified map, the local environment map and the several local internal maps are all the input scales of the urban multimodal large model.

[0016] A further preferred technical solution is: in S2, the process of compressing the original image with the small target object detection frame according to the input scale of the urban multimodal large model is:

[0017] First, calculate the compression factor of the long side of the original image to the input scale of the urban multimodal large model;

[0018] The long and short sides of the original image are proportionally compressed according to the compression factor, and the corresponding compressed image is obtained after proportional compression. It is determined whether the input scale of the compressed image meets the input scale of the urban multimodal large model. If so, the compressed image is used as a thumbnail and input into the urban multimodal large model. If not, the part of the compressed image with insufficient input scale is obtained based on the input scale of the compressed image and the input scale of the urban multimodal large model, and blank pixels are filled in this part to obtain a thumbnail, which is then input into the urban multimodal large model.

[0019] A further preferred technical solution is: the step S3 includes the following steps:

[0020] S31. Compare the input scales of all sides of the small target object detection frame in the original image with the input scale of the urban multimodal large model. If the input scale of any side of the small target object detection frame is smaller than the input scale of the urban multimodal large model, proceed to step S32; if the input scale of any side of the small target object detection frame is larger than the input scale of the urban multimodal large model, proceed to step S33.

[0021] S32. Cropping the original image with the small target object detection frame using the center point of the small target object detection frame, using the center point of the small target object detection frame as the center point of the cropping, and using the input scale of the urban multimodal large model as the cropping size, to obtain a corresponding local environment map;

[0022] S33. Compress the original image with the small target object detection frame using the center point of the small target object detection frame as the center point of compression. Compress the original image until the input scale of all sides of the small target object detection frame is less than or equal to the input scale of the urban multimodal large model, and obtain the corresponding local environment map.

[0023] A further preferred technical solution is: in S4, the traversal cutting refers to starting from the upper left corner of the small target object detection frame on the original image with the small target object detection frame, and cutting the original image with the small target object detection frame into blocks from left to right and then from top to bottom.

[0024] A further preferred technical solution is: the step S5 includes the following steps:

[0025] S51, inputting the simplified image, the local environment image, and several local internal images into the urban multimodal large model, and obtaining corresponding simplified image embedding features, local environment image embedding features, and local internal image embedding features through the urban multimodal large model;

[0026] S52, adding image spacing marks before and after the simplified image embedding feature, the local environment image embedding feature, and the local internal image embedding feature, and then sequentially splicing them to form a visual embedding feature.

[0027] A further preferred technical solution is: in S6, the process of embedding the instruction text features obtained after processing the instruction text is: inputting the instruction text into the large language model word segmenter and the text embedding model in sequence for processing, and obtaining the instruction text embedding features after processing.

[0028] A small target object detection system based on a large multimodal urban model, applying the small target object detection method based on a large multimodal urban model, comprises:

[0029] Urban multimodal large model module: obtains the original image with the small target object detection frame and the corresponding instruction text, and inputs the original image with the small target object detection frame into the urban multimodal large model;

[0030] Thumbnail acquisition module: compresses the original image with small target object detection boxes according to the input scale of the urban multimodal large model to obtain the corresponding thumbnail;

[0031] Local environment map acquisition module: This module crops or compresses the original image with the small target object detection frame according to the input scale of the urban multimodal large model and the input scale of the small target object detection frame to obtain the corresponding local environment map;

[0032] Local internal map acquisition module: On the original image with the small target object detection frame, traverse and cut from the upper left corner of the small target object detection frame as the starting point to obtain several corresponding local internal maps;

[0033] Visual embedding feature acquisition module: This module inputs the simplified image, local environment image, and several local internal images into the urban multimodal large model, and obtains visual embedding features with image interval labels through the urban multimodal large model;

[0034] Predicted text embedding feature acquisition module: obtains the instruction text embedding feature obtained after processing the instruction text, and obtains the predicted text based on the visual embedding feature with image interval marks and the instruction text embedding feature.

[0035] Beneficial effects of the present invention:

[0036] The present invention provides a small target object detection method and system based on a large multimodal urban model. The small target object detection method mainly considers that the detection target is relatively small relative to the original image. Therefore, the original image containing the small target object detection frame needs to be selectively processed differently. First, the original image is compressed considering the input scale of the large multimodal model to obtain a simplified image. Then, the original image is compressed or cropped considering the input scale of the large multimodal model and the input scale of the small target object detection frame to obtain a local environment map. Thus, the local environment map is traversed and cut to obtain multiple local internal maps, which can ensure that the small target object information is not lost, facilitating its detection by the large multimodal model.

[0037] In addition, image interval markers are added before and after the simplified image embedding features, local environment image embedding features, and local internal image embedding features, and they are sequentially spliced ​​to form visual embedding features with image interval markers. The visual embedding features can improve the efficiency of small target detection in multimodal large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1Schematic diagram of the steps of the small target object detection method in Example 1 of the present invention;

[0039] Figure 2 Schematic diagram of forming visual embedding features with image interval marks in the small target object detection method in Example 1 of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] Unless otherwise specifically stated, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0042] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0043] Additionally, descriptions of well-known structures, functions, and configurations may be omitted for clarity and conciseness. Those skilled in the art will recognize that various changes and modifications can be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0044] Technologies, methods and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods and equipment should be considered part of the authorization specification.

[0045] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0046] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments:

[0047] Example 1

[0048] In smart cities, cameras are usually installed on transportation equipment such as buses to collect information about the urban environment in real time. Multiple city street view pictures are collected, and then image analysis is performed on the city street view pictures to obtain the environmental characteristics captured in the current city street view pictures. Different action measures can be taken based on the environmental characteristics to achieve urban environment governance.

[0049] In the prior art, a multimodal large model is usually used to detect the collected urban street view images to obtain corresponding predicted text embedding features. The specific detection process is as follows: when information about the urban environment is collected by cameras mounted on transportation equipment such as buses, a data set consisting of multiple urban street view images is obtained; then, the urban street view images and the corresponding instruction texts are input into the multimodal large model, and in the multimodal large model, the urban street view images are sequentially input into the visual processing module and the visual-to-text space converter, and corresponding data processing is performed in the visual processing module and the visual-to-text space converter to obtain corresponding image embedding features; at the same time, the instruction text is sequentially input into the large language model word segmenter and the text embedding model for corresponding data processing to obtain corresponding instruction text embedding features; finally, the image embedding features and the text embedding features are input into the large language model, and the large language model is used for recognition to output corresponding predicted text embedding features.

[0050] The visual processing module refers to the VIT in the multimodal large model, and its basic principle is:

[0051] Image segmentation: The input image is divided into a series of fixed-size patches (patches), such as the common 16×16 size. Each image patch is flattened into a one-dimensional vector and then mapped to a high-dimensional vector space through a linear projection layer to form an embedded representation of the image patch. This process is similar to word embedding in natural language processing, converting the image into a "sequence" form suitable for Transformer processing.

[0052] In urban governance, when using multimodal large models to detect collected urban street view images, small target detection tasks are often encountered, such as scattered garbage, scattered garbage, posted advertisements, etc., that is, detecting small targets in urban street view images can be used to detect whether there is scattered garbage on urban roads.

[0053] As for the above-mentioned small target detection task, in the existing technology, when a multimodal large model is used to detect the collected urban street scene images, the general practice is: first, the original image is compressed to the input scale (448x448), then the entire original image is traversed, and the original image is evenly divided into multiple local segmented images. Finally, the original image and multiple local segmented images are input into the visual processing module of the multimodal large model in turn for processing, and the image embedding features are obtained through the visual to text space converter. At this time, due to the small proportion of the detection target relative to the original image, this practice of evenly dividing the original image and processing all the local segmented images reduces the operating efficiency of the multimodal large model. In particular, for today's ultra-high-resolution (4K) urban governance snapshot scenes, too many local segmented images may cause the tokens of the local image to overwhelm the tokens of the overall image, and even too many tokens may exceed the context length of the large model.

[0054] In order to solve the problem of low efficiency of the common practice of using multimodal large models for small target detection tasks in the prior art, the common practice of using multimodal large models in the prior art to detect urban street scene images collected is improved, and an urban multimodal large model is proposed. A new method is used to process the original image in the urban multimodal large model. When a small target object detection frame suspected to be a small target object appears in the original image, the original image can be processed accordingly according to the input scale of the urban multimodal large model. The simplified image, local environment image and local internal image obtained after processing are sequentially spliced ​​through image interval marks to form a visual embedding feature with image interval marks. The visual embedding feature can improve the efficiency of the multimodal large model for small target detection, and is more suitable for small target detection tasks in urban governance scenarios. It can be seen that the present invention performs relative detection on the original image with a small target object detection frame, and its purpose is to make the output small target information more comprehensive, which can reduce the false detection of the detection model.

[0055] like Figure 1 As shown, the small target object detection method includes the following steps:

[0056] S1. Obtain an original image with a small target object detection frame and the corresponding instruction text, and input the original image with the small target object detection frame into the urban multimodal large model;

[0057] S2. Compress the original image with the small target object detection frame according to the input scale of the urban multimodal large model to obtain the corresponding simplified image;

[0058] S3. Cropping or compressing the original image with the small target object detection frame according to the input scale of the urban multimodal large model and the input scale of the small target object detection frame to obtain a corresponding local environment map;

[0059] S4, performing traversal cutting on the original image with the small target object detection frame starting from the upper left corner of the small target object detection frame to obtain several corresponding local internal images;

[0060] S5. Inputting the simplified image, the local environment image, and several local internal images into the urban multimodal large model, and obtaining visual embedding features with image interval marks through the urban multimodal large model;

[0061] S6. Obtaining instruction text embedding features obtained after processing the instruction text, and obtaining predicted text embedding features based on the visual embedding features with the image spacing mark and the instruction text embedding features. In S6, first, the predicted text embedding features are obtained using the visual embedding features with the image spacing mark and the instruction text embedding features, and then the predicted text is obtained using the predicted text embedding features.

[0062] A further preferred technical solution is: the original picture with the small target object detection frame refers to the one obtained by inputting the collected city street view picture into the target recognition model for target recognition, and the small target object detection frame displayed on the original picture is obtained by target recognition. Specifically, the target recognition model can be a YOLO model, etc. Specifically, it can be a city target detection yolov5 model, a city target detection yolov6 model, or a city target detection yolov7 model. Through the target recognition model, the collected city street view picture is subjected to target recognition to obtain the small target object detection frame. In addition, the instruction text refers to the text manually input into the city street view picture, and the text is associated with the city street view picture as the input of the city multimodal large model.

[0063] A further preferred technical solution is that the scales of the simplified map, the local environment map and the several local internal maps are all the input scales of the urban multimodal large model.

[0064] A further preferred technical solution is: in S2, the process of compressing the original image with the small target object detection frame according to the input scale of the urban multimodal large model is:

[0065] First, calculate the compression factor of the long side of the original image to the input scale of the urban multimodal large model;

[0066] The long and short sides of the original image are proportionally compressed according to the compression factor, and the corresponding compressed image is obtained after proportional compression. It is determined whether the input scale of the compressed image meets the input scale of the urban multimodal large model. If so, the compressed image is used as a thumbnail and input into the urban multimodal large model. If not, the part of the compressed image with insufficient input scale is obtained based on the input scale of the compressed image and the input scale of the urban multimodal large model, and blank pixels are filled in this part to obtain a thumbnail, which is then input into the urban multimodal large model.

[0067] A further preferred technical solution is: the step S3 includes the following steps:

[0068] S31. Compare the input scales of all sides of the small target object detection frame in the original image with the input scale of the urban multimodal large model. If the input scale of any side of the small target object detection frame is smaller than the input scale of the urban multimodal large model, proceed to step S32; if the input scale of any side of the small target object detection frame is larger than the input scale of the urban multimodal large model, proceed to step S33.

[0069] S32. Cropping the original image with the small target object detection frame using the center point of the small target object detection frame, using the center point of the small target object detection frame as the center point of the cropping, and using the input scale of the urban multimodal large model as the cropping size, to obtain a corresponding local environment map;

[0070] S33. Compress the original image with the small target object detection frame using the center point of the small target object detection frame as the center point of compression. Compress the original image until the input scale of all sides of the small target object detection frame is less than or equal to the input scale of the urban multimodal large model, and obtain the corresponding local environment map.

[0071] A further preferred technical solution is: in S4, the traversal cutting refers to starting from the upper left corner of the small target object detection frame on the original image with the small target object detection frame, and cutting the original image with the small target object detection frame into blocks from left to right and then from top to bottom.

[0072] A further preferred technical solution is: the step S5 includes the following steps:

[0073] S51, inputting the simplified image, the local environment image, and several local internal images into the urban multimodal large model, and obtaining corresponding simplified image embedding features, local environment image embedding features, and local internal image embedding features through the urban multimodal large model;

[0074] S52, adding image spacing marks before and after the simplified image embedding feature, the local environment image embedding feature, and the local internal image embedding feature, and then sequentially splicing them to form a visual embedding feature.

[0075] A further preferred technical solution is: in S6, the process of embedding the instruction text features obtained after processing the instruction text is: inputting the instruction text into the large language model word segmenter and the text embedding model in sequence for processing, and obtaining the instruction text embedding features after processing.

[0076] In summary, for the small target detection task, when the urban multimodal large model is used to detect the collected urban street scene pictures in the small target detection method proposed in the present invention, the small target object detection method is as follows: first, the original picture with the small target object detection frame is compressed according to the input scale of the urban multimodal large model to obtain a thumbnail; then, the original picture with the small target object detection frame is cropped or compressed according to the input scale of the urban multimodal large model and the input scale of the small target object detection frame to obtain a local environment map; then, the original picture with the small target object detection frame is compressed to the input scale, and then the entire original picture is traversed, starting from the upper left corner of the small target object detection frame, and the original picture with the small target object detection frame is divided into blocks from left to right and then from top to bottom to obtain several local internal maps; finally, the thumbnail, the local environment map and the multiple local internal maps are input into the visual processing module in the urban multimodal large model for processing, and the corresponding thumbnail embedding features, local environment map embedding features and local internal map embedding features are obtained through the visual to text space converter. In addition, for the simple image embedding feature, local environment image embedding feature and local internal image embedding feature, the present invention also proposes image spacing marks, adding image spacing marks before and after the simple image embedding feature, local environment image embedding feature and local internal image embedding feature, and then splicing them in sequence to form a visual embedding feature with image spacing marks.

[0077] Specifically, such as Figure 2 As shown, the small target object detection method processes the original image with a small target detection frame through three different processing methods to obtain a thumbnail image, a local environment image and multiple local internal images, and then the thumbnail image, the local environment image and the local internal image are processed by a visual processing module to obtain corresponding thumbnail image embedding features, local environment image embedding features and local internal image embedding features, and image spacing marks are added before and after the thumbnail image embedding features, the local environment image embedding features and the local internal image embedding features, and then they are sequentially spliced ​​to form visual embedding features with image spacing marks, and the visual embedding features with image spacing marks and the corresponding instruction text embedding features are input into the large language model, and the predicted text embedding features are obtained through the large language model, which can improve the efficiency of small target detection.

[0078] Example 2

[0079] A small target object detection system based on a large multimodal urban model, applying the small target object detection method based on a large multimodal urban model, comprises:

[0080] Urban multimodal large model module: obtains the original image with the small target object detection frame and the corresponding instruction text, and inputs the original image with the small target object detection frame into the urban multimodal large model;

[0081] Thumbnail acquisition module: compresses the original image with small target object detection boxes according to the input scale of the urban multimodal large model to obtain the corresponding thumbnail;

[0082] Local environment map acquisition module: This module crops or compresses the original image with the small target object detection frame according to the input scale of the urban multimodal large model and the input scale of the small target object detection frame to obtain the corresponding local environment map;

[0083] Local internal map acquisition module: On the original image with the small target object detection frame, traverse and cut from the upper left corner of the small target object detection frame as the starting point to obtain several corresponding local internal maps;

[0084] Visual embedding feature acquisition module: This module inputs the simplified image, local environment image, and several local internal images into the urban multimodal large model, and obtains visual embedding features with image interval labels through the urban multimodal large model;

[0085] Predicted text embedding feature acquisition module: obtains the instruction text embedding feature obtained after processing the instruction text, and obtains the predicted text based on the visual embedding feature with image interval marks and the instruction text embedding feature.

[0086] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Based on the technical essence of the present invention and within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement of the above embodiment shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A small target object detection method based on a large multimodal urban model, characterized in that: The small target object detection method comprises the following steps: S1. Obtain an original image with a small target object detection frame and the corresponding instruction text, and input the original image with the small target object detection frame into the urban multimodal large model; S2. Compress the original image with the small target object detection frame according to the input scale of the urban multimodal large model to obtain the corresponding simplified image; S3. Cropping or compressing the original image with the small target object detection frame according to the input scale of the urban multimodal large model and the input scale of the small target object detection frame to obtain a corresponding local environment map; S4, performing traversal cutting on the original image with the small target object detection frame starting from the upper left corner of the small target object detection frame to obtain several corresponding local internal images; S5. Inputting the simplified image, the local environment image, and several local internal images into the urban multimodal large model, and obtaining visual embedding features with image interval marks through the urban multimodal large model; S6. Obtain instruction text embedding features obtained after processing the instruction text, and obtain predicted text based on the visual embedding features with image interval marks and the instruction text embedding features.

2. The small target object detection method based on a large multimodal urban model according to claim 1 is characterized in that: The original image with the small target object detection frame refers to the image obtained by inputting the collected city street view image into the target recognition model for target recognition, and the small target object detection frame displayed on the original image is obtained through target recognition.

3. The small target object detection method based on a large multimodal urban model according to claim 1 is characterized in that: The scales of the simplified map, the local environment map and the several local internal maps are all input scales of the urban multimodal large model.

4. The small target object detection method based on a large multimodal urban model according to claim 1 is characterized in that: In S2, the process of compressing the original image with small target object detection boxes according to the input scale of the urban multimodal large model is as follows: First, calculate the compression factor of the long side of the original image to the input scale of the urban multimodal large model; The long and short sides of the original image are proportionally compressed according to the compression factor, and the corresponding compressed image is obtained after proportional compression. It is determined whether the input scale of the compressed image meets the input scale of the urban multimodal large model. If so, the compressed image is used as a thumbnail and input into the urban multimodal large model. If not, the part of the compressed image with insufficient input scale is obtained based on the input scale of the compressed image and the input scale of the urban multimodal large model, and blank pixels are filled in this part to obtain a thumbnail, which is then input into the urban multimodal large model.

5. The small target object detection method based on a large multimodal urban model according to claim 1 is characterized in that: The step S3 includes the following steps: S31. Compare the input scales of all sides of the small target object detection frame in the original image with the input scale of the urban multimodal large model. If the input scale of any side of the small target object detection frame is smaller than the input scale of the urban multimodal large model, proceed to step S32; if the input scale of any side of the small target object detection frame is larger than the input scale of the urban multimodal large model, proceed to step S33. S32. Cropping the original image with the small target object detection frame using the center point of the small target object detection frame, using the center point of the small target object detection frame as the center point of the cropping, and using the input scale of the urban multimodal large model as the cropping size, to obtain a corresponding local environment map; S33. Compress the original image with the small target object detection frame using the center point of the small target object detection frame as the center point of compression. Compress the original image until the input scale of all sides of the small target object detection frame is less than or equal to the input scale of the urban multimodal large model, and obtain the corresponding local environment map.

6. The small target object detection method based on a large multimodal urban model according to claim 1, characterized in that: In S4, the traversal cutting refers to cutting the original image with the small target object detection frame into blocks from left to right and then from top to bottom, starting from the upper left corner of the small target object detection frame.

7. The small target object detection method based on a large multimodal urban model according to claim 1 is characterized in that: The step S5 includes the following steps: S51, inputting the simplified image, the local environment image, and several local internal images into the urban multimodal large model, and obtaining corresponding simplified image embedding features, local environment image embedding features, and local internal image embedding features through the urban multimodal large model; S52, adding image spacing marks before and after the simplified image embedding feature, the local environment image embedding feature, and the local internal image embedding feature, and then sequentially splicing them to form a visual embedding feature.

8. The small target object detection method based on a large multimodal urban model according to claim 1 is characterized in that: In S6, the process of embedding the instruction text features obtained after processing the instruction text is as follows: the instruction text is sequentially input into the large language model word segmenter and the text embedding model for processing, and the instruction text embedding features are obtained after processing.

9. A small target object detection system based on a large multimodal urban model, characterized in that: The method for detecting small targets based on a large multimodal urban model according to any one of claims 1 to 8 comprises: Urban multimodal large model module: obtains the original image with the small target object detection frame and the corresponding instruction text, and inputs the original image with the small target object detection frame into the urban multimodal large model; Thumbnail acquisition module: compresses the original image with small target object detection boxes according to the input scale of the urban multimodal large model to obtain the corresponding thumbnail; Local environment map acquisition module: This module crops or compresses the original image with the small target object detection frame according to the input scale of the urban multimodal large model and the input scale of the small target object detection frame to obtain the corresponding local environment map; Local internal map acquisition module: On the original image with the small target object detection frame, traverse and cut from the upper left corner of the small target object detection frame as the starting point to obtain several corresponding local internal maps; Visual embedding feature acquisition module: This module inputs the simplified image, local environment image, and several local internal images into the urban multimodal large model, and obtains visual embedding features with image interval labels through the urban multimodal large model; Predicted text embedding feature acquisition module: obtains the instruction text embedding feature obtained after processing the instruction text, and obtains the predicted text based on the visual embedding feature with image interval marks and the instruction text embedding feature.

Images