Smart city perception image processing system and method carrying multi-modal model

Abstract

The invention discloses a smart city perception image processing system and method carrying a multi-modal model, and the system comprises an original visual coding structure, an enhanced visual coding structure, a special visual coding structure, a visual group fusion and compression module, a multi-layer perception MLPs structure, and a large language model. Performing various visual coding on the original reasoning picture, and then outputting original visual coding features, enhanced visual coding features and special visual coding features with consistent feature dimensions; and fusing and compressing each visual coding feature, mapping the multi-level image feature to a language semantic space to obtain an input image feature, splicing the input image feature and the text feature, and generating a language response output. A special visual feature extraction module is provided for a city management scene, the perception ability of the system to a specific scene can be improved, and thus the accuracy of city perception information is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a smart city perception image processing system and method equipped with a multimodal model. Background Art

[0002] In the smart city AI supervision system, event judgment based on accurate description of target attributes is crucial. This description can not only effectively distinguish the differences in the attributes of the same target, but also provide detailed event judgment for the smart city.

[0003] Existing smart city AI detection systems achieve target detection through object detection models. Typically, these devices employ CNN-based Yolo models or transformer-based Detr models. These systems leverage scene data to detect urban objects and then use their corresponding categories or add certain logic to determine events. However, existing object detection systems only provide target categories, not target attributes. Furthermore, relying solely on large amounts of data and a single target detection model struggles to address two key challenges in smart city object classification prediction and abstract representation. First, image detection models lacking abstract textual features cannot effectively distinguish between different target categories for the same object in different scenarios, such as dirty roads versus dirty green spaces. Second, lacking textual representation of target attributes hinders event recognition and judgment. For example, a detection model cannot perceive the lane or lane location of a target vehicle, making it impossible to distinguish between urban management events (parking on the lane and in the lane are distinct events). While existing approaches employing a combination of multiple visual models to address smart city management issues can improve effectiveness to a certain extent, they fail to fundamentally address urban management issues and, in fact, increase operational and maintenance costs due to complex logic. While general multimodal big models can be used to solve urban management problems, they cannot fully express the visual characteristics of complex smart city scenes, making it difficult to implement vertical multimodal big models for smart cities. Even if multimodal big models are used to achieve target attributes and even more inference content, the inadequate expression of visual features in multimodal big models results in insufficient representation of image-related target attributes or inference content. Summary of the Invention

[0004] The present invention aims to provide a smart city perception image processing system and method based on a multimodal model. Based on the LLaVA multimodal large model, a hybrid visual feature encoding group is constructed. While retaining the original visual feature extraction, additional enhanced visual encoding is added to further improve the generalization capability of the visual features. Furthermore, a proprietary visual encoding structure is provided for urban management scenarios, improving the accuracy of urban perception information.

[0005] To achieve the above object, the present invention proposes the following solutions:

[0006] In one aspect, the present invention provides a smart city perception image processing system equipped with a multimodal model, comprising:

[0007] Hybrid visual feature coding group, the hybrid visual feature coding group includes three visual coding structures with different functions, namely:

[0008] The original visual coding structure is used to receive the original inference image for original visual coding and output the original visual coding features;

[0009] Enhanced visual coding structure, which is used to receive the original inference image, perform enhanced image coding, and output enhanced visual coding features;

[0010] A proprietary visual coding structure is used to receive the original inference image, perform proprietary visual image coding, and convert it, outputting proprietary visual coding features that are consistent with the dimensions of the original visual coding features and enhanced visual coding features;

[0011] The visual group fusion and compression module is used to fuse and compress the original visual coding features, enhanced visual coding features, and proprietary visual coding features to output multi-scale image features;

[0012] Multi-layer perception (MLPs) structure is used to map multi-scale image features to language semantic space to obtain input image features;

[0013] The large language model is used to splice the input image features and the text features obtained based on the text input instructions corresponding to the original inference image to generate a language response output.

[0014] In some specific embodiments, the proprietary visual encoding structure includes: a Yolov11 network structure, a scene-specific feature extraction module and a decision module, wherein:

[0015] The decision module is used to perform a binary classification decision on the received raw reasoning image to determine whether further scene-specific features need to be extracted. If the judgment result is yes, the raw reasoning image is input into the scene-specific feature extraction module and the Yolov11 network structure respectively. If the judgment result is no, the raw reasoning image is directly input into the Yolov11 network structure;

[0016] The scene-specific feature extraction module is used to extract the specific features of the original inference image and input the first feature into the backbone network of the Yolov11 network structure;

[0017] The Yolov11 network structure includes a backbone network, a neck network and a head network. The backbone network is embedded with a scene-specific feature extraction module, and decides whether to fuse it with the first feature output by the scene-specific feature extraction module based on the judgment result of the decision module.

[0018] In some specific embodiments, the four-layer structure of the scene-specific feature extraction module is: two layers of identical Conv modules connected in series, one layer of C3K2 modules, and a last layer of Conv modules, wherein the two layers of identical Conv modules connected in series and one layer of C3K2 modules are used to extract proprietary visual features from the original reasoning image, and the last layer of Conv module is used to match the proprietary visual features with the feature dimensions of the Yolov11 network structure and output the first feature.

[0019] In some specific implementation schemes, the first three layers of the scene-specific feature extraction module are the same as the first three layers of the backbone network of the Yolov11 network structure, and the scene-specific feature extraction module is embedded between the first three layers and the fourth layer of the backbone network of the Yolov11 network structure.

[0020] In some specific implementation schemes, the size of the convolution kernel in the last layer of the Conv module structure is 1, the step size of the convolution operation is 1, and the last layer of the Conv module adjusts the dimension of the proprietary visual feature to be consistent with the second feature dimension output by the C3K2 module of the backbone network, outputs the first feature, and then uses the cat structure to fuse the first feature and the second feature.

[0021] In some specific embodiments, the visual group fusion and compression module includes a feature linking layer, a compression linear layer, a normalization function, and an activation layer, wherein the feature linking layer is used to concatenate and fuse the original visual coding features, the enhanced visual coding features, and the proprietary visual coding features to obtain a first image feature;

[0022] The compressed linear layer is used to perform feature compression and effective feature extraction on the first image feature to obtain effective features;

[0023] The normalization function and activation layer are used to normalize and linearly transform the effective features respectively to obtain multi-scale image features.

[0024] In some specific embodiments, the multi-layer perception MLPs structure includes a multi-layer perceptron MLP module with multiple cycles and residual connections. The multi-layer perceptron MLP module includes a first fully connected layer, an activation function, and a second fully connected layer connected in sequence. The input multi-scale image features are subjected to multiple cycles and then are residually connected with the output from the second fully connected layer to output the input image features.

[0025] In a second aspect, the present application provides a method for processing smart city perception images using a multimodal model, comprising the following steps:

[0026] S1. Input the city perception image as the original inference image into the trained hybrid visual feature coding group for image feature coding. The hybrid visual feature coding group includes the original visual coding structure, the enhanced visual coding structure, and the proprietary visual coding structure, and outputs the original visual coding features, the enhanced visual coding features, and the proprietary visual coding features respectively.

[0027] S2. Input the original visual coding features, enhanced visual coding features, and proprietary visual coding features into the visual group fusion and compression module for row fusion and compression, and output multi-scale image features;

[0028] S3, input the multi-scale image features into the multi-layer perception MLPs structure for mapping to obtain input image features;

[0029] S4, performing a text description on the city perception image, obtaining a corresponding text input instruction, inputting the text input instruction into a text instruction encoding structure, converting the text input instruction, and obtaining text features;

[0030] S5. Input the input image features and text features into the trained large language model, concatenate the input image features and text features, and generate language response output.

[0031] In some specific embodiments, the proprietary visual encoding structure includes a Yolov11 network structure, a scene-specific feature extraction module, and a decision module. The training process of the proprietary visual encoding structure includes the following steps:

[0032] S11. Construct a training data set, collect several perception pictures related to urban management and store them in the training data set, recorded as data1;

[0033] S12, filter out the perception images corresponding to the proprietary scene from data1 and label these perception images with "yes" and save them in a proprietary image dataset, recorded as data2, label the perception images of the difference set of data1 and data2 with "no", and then combine the labeled data2 and the difference set into an image decision label dataset, recorded as data3;

[0034] S13, the training is divided into auxiliary decision training, first stage training and second stage training. The auxiliary decision training is to use data3 to train the decision module to obtain the optimal binary target detection decision model and the optimal weight weight3 of the model;

[0035] S14, the first stage of training is to use data1 to train the Yolov11 network structure to obtain the best target detection model and its weight weight1;

[0036] S15, the second stage of training is to use data2 to train the scene-specific feature extraction module to obtain the optimal extraction model and its weight weight2;

[0037] S16. According to weight1, weight2, and weight3, the weights of the Yolov11 network structure, the scene-specific feature extraction module, and the decision module are adjusted respectively to obtain a trained proprietary visual encoding structure.

[0038] In some specific embodiments, the decision module selects the resnet50 classification model.

[0039] The present invention has the beneficial effects:

[0040] Based on the LLaVA multimodal large model, the present invention constructs a visual feature coding group. In addition to retaining the original visual feature extraction, it adds additional enhanced visual coding to further improve the generalization ability of its visual features. In addition, a proprietary visual feature extraction module is provided for urban management scenarios. This module is pre-trained in existing scenarios to obtain proprietary features adapted to the scenarios. The proprietary visual coding structure can improve the system's perception of specific scenarios, thereby improving the accuracy of urban perception information, helping city managers better perceive real events in the city and facilitate urban management. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of the composition of a smart city perception image processing system equipped with a multimodal model proposed in this application;

[0042] Figure 2 A schematic diagram of the composition of a proprietary visual coding structure provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of the composition of the visual group fusion and compression module provided in an embodiment of the present invention;

[0044] Figure 4 A diagram showing the system composition of the LLaVA multimodal large model provided by an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of the structure of the multi-layer perception MLPs provided in an embodiment of the present invention;

[0046] Figure 6 A schematic diagram of the proprietary visual feature extraction training process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0048] Unless otherwise specifically stated, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0049] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0050] Additionally, descriptions of well-known structures, functions, and configurations may be omitted for clarity and conciseness. Those skilled in the art will recognize that various changes and modifications can be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0051] Technologies, methods and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods and equipment should be considered part of the authorization specification.

[0052] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0053] Example 1

[0054] like Figure 1 As shown, this embodiment provides a smart city perception image processing system equipped with a multimodal model. The entire system is improved on the basis of the LLaVA multimodal large model, including:

[0055] Hybrid visual feature coding group, the hybrid visual feature coding group includes three visual coding structures with different functions, namely:

[0056] The original visual coding structure is used to receive the original inference image for original visual coding and output the original visual coding features;

[0057] Enhanced visual coding structure, which is used to receive the original inference image, perform enhanced image coding, and output enhanced visual coding features;

[0058] A proprietary visual coding structure is used to receive the original inference image, perform proprietary visual image coding, and convert it, outputting proprietary visual coding features that are consistent with the dimensions of the original visual coding features and enhanced visual coding features;

[0059] The visual group fusion and compression module is used to fuse and compress the original visual coding features, enhanced visual coding features, and proprietary visual coding features to output multi-scale image features;

[0060] Multi-layer perception (MLPs) structure is used to map multi-scale image features to language semantic space to obtain input image features;

[0061] The large language model is used to splice the input image features and the text features obtained based on the text input instructions corresponding to the original inference image to generate a language response output.

[0062] Specifically, the raw inference images referred to in this application refer to images of various urban events captured by cameras or urban surveillance equipment mounted on mobile vehicles (such as city buses). For example, these vehicles can capture images of peddlers on city roads. In this context, peddlers are vehicles that stop on various city roads to sell goods, such as fruits, vegetables, and snacks. They can use four-wheeled trucks, three-wheeled trucks, or other cargo-carrying vehicles like three-wheeled bicycles. Their characteristic characteristic is that these vehicles, carrying goods for sale, remain in a certain location for a long period of time. In the context of peddlers, accurately detecting whether peddlers are occupying the road at their location through images would help city managers manage peddlers and prevent them from occupying the road. However, due to the complexity of peddlers, misidentification is prone to occur. For example, a vehicle carrying goods but not selling goods while in motion, simply transporting goods, could be mistakenly identified as a peddler. Alternatively, vehicles such as fruit carts temporarily parked on the road could be mistakenly identified as peddlers. In view of this situation, the prior art uses target detection to identify whether the person is a traveling peddler, but it can only provide the target category, and complex scenes cannot be distinguished or even provide wrong judgments. For this reason, it is very necessary to adopt a text description and reasoning ability that takes into account scene factors or a large model with attribute text description. Therefore, this application uses a multimodal large model to solve these problems. In addition, in order to improve the recognition accuracy, this application also proposes to improve the existing Yolov11 network structure and add a scene-specific feature extraction module, which can optimize the problem of inaccurate processing of perceived images obtained in specific scenes, such as in bad weather, foggy days or strong light, and improve the processing accuracy of the model for pictures taken in these specific scenes.

[0063] The following is an explanation of each of the above modules:

[0064] 1. Hybrid visual feature encoding group

[0065] 1.1. Enhanced visual encoding module or original visual encoding structure

[0066] The enhanced visual coding module or the original visual coding structure basically uses a VIT structure model, which in turn uses a transformer-based structure. For example, the enhanced visual coding structure uses the SAM2 model. The existing transformer structure proposes an encoder-decoder structure, while the underlying structure uses a transformer and FFN structure. As can be seen from the figure, the transformer structure consists of Q, K, and V, where Q, K, and V are implemented using conventional linear convolution. The transformer structure is finally implemented using the following formula 1, which is calculated as follows:

[0067]

[0068] where d k Represents the feature expression dimension.

[0069] The FFN structure consists of two linear convolutions and a ReLU activation function, and its calculation formula is as follows:

[0070] FFN(x)=max(0,xW1+b1)w2+b2 (2)

[0071] 1.2 Proprietary Visual Encoding Structure

[0072] The proprietary visual encoding structure is an improvement on the Yolov11 network structure. It is used to address conditions such as lighting and defogging. It is a method for obtaining proprietary visual features for urban management. This is a feature information expression that the multimodal and enhancement modules do not have. Specifically, it includes:

[0073] 1.2.1 Decision Module

[0074] The decision module is used to make a binary classification decision on the received original reasoning image to determine whether it is necessary to further extract scene-specific features. If the judgment result is yes, the original reasoning image is input into the scene-specific feature extraction module and the Yolov11 network structure respectively. If the judgment result is no, the original reasoning image is directly input into the Yolov11 network structure. The decision module uses the resnet50 classification model as the decision model.

[0075] 1.2.2 Scenario-specific Feature Extraction Module

[0076] The scene-specific feature extraction module is used to extract the specific features of the original inference image and input the first feature into the backbone network of the Yolov11 network structure; Figure 2As shown, the four-layer structure of the scene-specific feature extraction module is: two identical layers of tandem Conv modules, a layer of C3K2 modules, and a final Conv module. The two identical layers of tandem Conv modules and the C3K2 module are used to extract specific visual features from the original inference image. The final Conv module is used to match the specific visual features with the feature dimensions of the Yolov11 network structure and output the first feature. The first three layers of the scene-specific feature extraction module are identical to the first three layers of the Yolov11 network backbone network. The scene-specific feature extraction module is embedded between the first three layers of the Yolov11 backbone network and the fourth layer.

[0077] In order to enhance the feature adaptability structure, in the last layer Conv module structure, Conv k=1, s=1 is the adjusted feature, the size of the convolution kernel is 1, and the step size of the convolution operation is 1. The last layer Conv module adjusts the dimension of the proprietary visual feature to be consistent with the second feature dimension output by the C3K2 module of the backbone network, outputs the first feature (if the decision module determines that the scene-specific feature extraction module needs to be used, the feature is output), and then uses the cat structure to directly add the first and second features for fusion.

[0078] 1.2.3Yolov11 Network Structure

[0079] The Yolov11 network structure in this embodiment is the same as the existing structure, except that a scene-specific feature extraction module is embedded between the first three layers and the fourth layer to accurately identify images in specific scenes, such as Figure 2 As shown in the figure, the Yolov11 network structure includes a backbone network (Backbone), a neck network (Neck) and a head network (Head), wherein the backbone network is embedded with a scene-specific feature extraction module, and decides whether to fuse the first feature output by the scene-specific feature extraction module based on the judgment result of the decision module.

[0080] 2. Visual group fusion and compression module (Concat+linear project)

[0081] like Figure 3As shown in the figure, the visual group fusion and compression module includes a feature link layer (concate), a compressed linear layer (nn.linear), a normalization function (BN function), and an activation layer (RELU). The feature link layer is used to concatenate and fuse the original visual coding features, enhanced visual coding features, and proprietary visual coding features to obtain the first image features; the compressed linear layer is used to perform feature compression and effective feature extraction on the first image features to obtain effective features; the normalization function and the activation layer are used to perform normalization and linear transformation on the effective features, respectively, to obtain multi-scale image features. The concate module is used to fuse the three visual features together, which is called concatenation; the nn.liear method is then used for feature compression, which can reduce high-dimensional features to low dimensions through linear mapping, remove redundant information, focus on key features, and meet the original LLaVA model feature format. For example, the 768-dimensional features output by ViT are compressed to 50 dimensions; and then normalization and activation functions are used.

[0082] Specifically, the original visual coding features are obtained from the original visual coding structure, and their format is (BLC); the enhanced visual coding features are obtained from the enhanced visual coding structure, and their format is (BL1C1); the proprietary visual coding features are obtained from the proprietary visual coding structure, and their format is (BC3H3W3). In order to make the proprietary visual coding features consistent with the formats of the original visual coding features and the enhanced visual coding features, an adaptive visual feature conversion structure is provided between the hybrid visual feature coding group and the visual group fusion and compression module, which converts the formats of the proprietary visual coding features, the original visual coding features, and the enhanced visual coding features into the same format. The adaptive visual feature conversion structure includes three MLP feature conversion modules, which process the proprietary visual coding features, the original visual coding features, and the enhanced visual coding features respectively. Among them, the proprietary visual coding features are converted by first converting (BC3H3W3) into (BC3H3*W3) through format conversion, and then performing position conversion (BH3*W3C3) to finally obtain (BL3C3). Where L3=H3*W3.

[0083] As can be seen, the output format of the original, enhanced, and proprietary visual coding structures is [BLC], where B represents the batch of input images, L represents the token, and C represents the vector representation of each token. However, the original LLaVA model image input requires BLC structured data. To address this, a fusion compression module is used to more effectively integrate the fused feature representations.

[0084] The specific steps of the compression process of the visual group fusion and compression module are as follows:

[0085] Perform cat connection on the (BLC), (BL1 C1), and (BL3 C3) obtained above. The three BLC structure data will be linked into one data, BL C+C1+C3, in the following way:

[0086] X=Concat(X o X e X p )

[0087] Xo, Xe, and Xp represent the original visual coding features, enhanced visual coding features, and proprietary visual coding features, respectively; X represents direct concatenation of C into 3C; therefore, the fusion uses direct concatenation to transform the three visual features into a matrix BL3C. However, the large language model requires the BLC format. Therefore, the project structure is then used to directly compress BL 3C into BLC using a compression module. It becomes (BL3C+C1+C3) and becomes feature F4. This step fuses multiple visual features. F4 is compressed through a linear layer structure, and the original feature F4's (BL3C+C1+C3) format becomes (BL3C). This step compresses the features and effectively extracts the features.

[0088] Normalization operation. Normalize the features to normalize the distribution of input data in each layer, speed up the training process, stabilize the neural network, and improve model performance.

[0089] Activation function operation. Introducing nonlinear transformations enables neural networks to learn and express complex patterns.

[0090] 3. Multi-layer perception MLPs structure

[0091] like Figure 5 As shown in the figure, the multi-layer perception MLPs structure is obtained by multiple cycles of a multi-layer perceptron MLP module, and an MLP structure with residual connection is added, which specifically includes two fully connected layers and an activation function, as shown in the formula below.

[0092] M(x)=x+[W2(σ(W1*x+b1))+b2]

[0093] Among them, σ(·) is the activation function, and this application chooses RELU as the activation function; W1 and W2 represent nn.Linear linear weights, respectively, and b1 and b2 are the corresponding biases. A multi-layer perception MLPs structure is adopted, so that the input multi-scale image features are passed through multiple cycles Nx and then connected with the output of the second fully connected layer through a residual connection to output the input image features that meet the input of the large language model. Since the spatial state and feature dimension of the multi-scale image features are higher than those of the text features, in order to enable the multi-scale image features and text features to be spliced ​​in the large language model, it is necessary to map the multi-scale image features to the language semantic space so that they are consistent with the state of the text features.

[0094] 4. Large Language Model

[0095] like Figure 4 As shown, in this embodiment, the entire image processing system is improved based on the LLaVA multimodal large model. The LLaVA multimodal large model adopts the CLIP visual feature network as the visual encoder. The multi-scale image feature XvImage after fusion and compression by the visual group fusion and compression module is converted into a language embedding token Hv in the word embedding space after passing through the multi-layer perception MLPs structure. The text input instruction Xq language instruction is converted into a text feature Hq through the text instruction encoding structure. Hv and Hq are passed through the original large language model Language Model of the LLaVA multimodal large model to generate a language response Language Response Xa output.

[0096] Example 2

[0097] With respect to the system proposed in Example 1, this embodiment provides a method for processing smart city perception images using a multimodal model, comprising the following steps:

[0098] S1. Input the city perception image as the original inference image into the trained hybrid visual feature coding group for image feature coding. The hybrid visual feature coding group includes the original visual coding structure, the enhanced visual coding structure, and the proprietary visual coding structure, and outputs the original visual coding features, the enhanced visual coding features, and the proprietary visual coding features respectively.

[0099] S2. Input the original visual coding features, enhanced visual coding features, and proprietary visual coding features into the visual group fusion and compression module for row fusion and compression, and output multi-scale image features;

[0100] S3, input the multi-scale image features into the multi-layer perception MLPs structure for mapping to obtain input image features;

[0101] S4, performing a text description on the city perception image, obtaining a corresponding text input instruction, inputting the text input instruction into a text instruction encoding structure, converting the text input instruction, and obtaining text features;

[0102] S5. Input the input image features and text features into the trained large language model, concatenate the input image features and text features, and generate language response output.

[0103] Specifically, the training process of each module of the system includes the following steps:

[0104] Step 1: Proprietary visual feature extraction training

[0105] S11. Construct a training data set, collect several perception pictures related to urban management and store them in the training data set, recorded as data1;

[0106] S12. Filter out the perception images corresponding to proprietary scenes (proprietary scenes include scenes such as bad weather, foggy days, or strong light) from data1 and label these perception images with "yes" and save them in a proprietary image dataset, recorded as data2. Label the perception images of the difference set of data1 and data2 with "no" and then combine the labeled data2 and the difference set into an image decision label dataset, recorded as data3.

[0107] S13, the training is divided into auxiliary decision training, first-stage training and second-stage training. The auxiliary decision training uses data3 to train the decision module. The decision module selects the resnet50 classification model to obtain the optimal binary target detection decision model and the optimal weight weight3 of the model. The optimal binary target detection decision model is recorded as model3.

[0108] S14, the first stage of training is to use data1 to train the Yolov11 network structure to obtain the best target detection model and its weight weight1, and the best target detection model is recorded as model1;

[0109] S15. Use the same module as Yolov11 to build a proprietary feature extraction module. This module has two Convs and one C3K2 for special processing of proprietary scenes. The last Conv is used to extract proprietary scene features that adapt to the original model feature space of Yolov11. Finally, use the cat structure to directly add them together for fusion. The second stage of training is to use data2 to train the scene-specific feature extraction module. At the same time, use weight1 to assign values ​​to the Yolov11 network structure during training. When training the scene-specific feature extraction module, freeze all structures of the Yolov11 network structure and only train the scene-specific feature extraction module to obtain the optimal extraction model, recorded as model2, and its corresponding optimal weight weight2;

[0110] S16. According to weight1, weight2, and weight3, the weights of the Yolov11 network structure, the scene-specific feature extraction module, and the decision module are adjusted respectively to obtain a trained proprietary visual encoding structure.

[0111] Step 2: Proprietary Visual Feature Extraction, Embedding, and Inference

[0112] The optimal models corresponding to the Yolov11 network structure, scene-specific feature extraction module, and decision module obtained above are embedded in the system. The specific process is as follows:

[0113] S21. Build a proprietary visual encoding structure. The integrated decision module + scene-specific feature extraction module is embedded in the Yolov11 network structure, such as Figure 6 As shown in the figure, the weights trained above are assigned to each structure to obtain a complete vision-specific feature extraction network. The original trained head module has been removed here, and the features are directly output in the format of (B C3 H3 W3).

[0114] S22, binary classification decision model reasoning. Input the reasoning image I and perform a binary classification decision to determine whether further extraction of specific visual features is required. If yes, the decision is yes; otherwise, the decision is no.

[0115] S23: Dedicated visual feature reasoning. If the decision in step S22 is yes, the inference image I is input into the scene-specific feature extraction module for reasoning, and the final output is the first feature F1; if the decision in step S22 is no, no reasoning is performed.

[0116] S24: Inference of proprietary visual feature embedding. Image I is processed through the first three layers of Yolov11 (Conv-Conv and C3K2) to obtain the second feature F2. If the decision in step S22 is yes, features F1 and F2 are directly added together to form the fused feature F3. If the decision in step S22 is no, feature F3 is equal to feature F2.

[0117] S25, model inference. After obtaining F3 in step S24, continue to complete the processing of the remaining structure of Yolov11, and finally obtain the proprietary visual encoding feature output, whose format is (B C3 H3 W3)

[0118] Step 3: Training a large language model

[0119] S31. Construct a dataset data4. The data in this dataset corresponds to the multimodal sample data annotation of urban management, which is an image and text annotation of multiple rounds of dialogue;

[0120] S32: Train the large language model. Input data4 into the large language model and perform model training to obtain the optimal model. While the large language model is training, the structure of the hybrid visual feature encoding group needs to be frozen, and no weight update is performed.

[0121] S33. Model deployment. The proprietary visual encoding structure obtained in step 1 is combined with the optimal model obtained in step S32 to obtain a large multimodal model. The large multimodal model is deployed to a cloud server and applied to smart city-related tasks.

[0122] It is understood that this application improves upon the LLaVA multimodal large model by constructing a visual feature encoding group. While retaining the original visual feature extraction, it also adds additional enhanced visual encoding to further improve the generalization of its visual features. Furthermore, a proprietary visual feature extraction module is provided for urban management scenarios. This module is pre-trained on existing scenarios to obtain scenario-adaptive proprietary features.

[0123] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Based on the technical essence of the present invention and within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement of the above embodiment shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A smart city perception image processing system equipped with a multimodal model, characterized in that: include: Hybrid visual feature coding group, the hybrid visual feature coding group includes three visual coding structures with different functions, namely: The original visual coding structure is used to receive the original inference image for original visual coding and output the original visual coding features; Enhanced visual coding structure, which is used to receive the original inference image, perform enhanced image coding, and output enhanced visual coding features; A proprietary visual coding structure is used to receive the original inference image, perform proprietary visual image coding, and convert it, outputting proprietary visual coding features that are consistent with the dimensions of the original visual coding features and enhanced visual coding features; The visual group fusion and compression module is used to fuse and compress the original visual coding features, enhanced visual coding features, and proprietary visual coding features to output multi-scale image features; Multi-layer perception (MLPs) structure is used to map multi-scale image features to language semantic space to obtain input image features; The large language model is used to splice the input image features and the text features obtained based on the text input instructions corresponding to the original inference image to generate a language response output.

2. The smart city perception image processing system equipped with a multimodal model according to claim 1, characterized in that: The proprietary visual encoding structure includes: Yolov11 network structure, scene-specific feature extraction module and decision module, among which, The decision module is used to perform a binary classification decision on the received raw reasoning image to determine whether further scene-specific features need to be extracted. If the judgment result is yes, the raw reasoning image is input into the scene-specific feature extraction module and the Yolov11 network structure respectively. If the judgment result is no, the raw reasoning image is directly input into the Yolov11 network structure; The scene-specific feature extraction module is used to extract the specific features of the original inference image and input the first feature into the backbone network of the Yolov11 network structure; The Yolov11 network structure includes a backbone network, a neck network and a head network. The backbone network is embedded with a scene-specific feature extraction module, and decides whether to fuse it with the first feature output by the scene-specific feature extraction module based on the judgment result of the decision module.

3. The smart city perception image processing system equipped with a multimodal model according to claim 2, characterized in that: The four-layer structure of the scene-specific feature extraction module is: two identical layers of Conv modules connected in series, one layer of C3K2 module, and the last layer of Conv module. The two identical layers of Conv modules connected in series and one layer of C3K2 module are used to extract proprietary visual features from the original inference image. The last layer of Conv module is used to match the proprietary visual features with the feature dimensions of the Yolov11 network structure and output the first feature.

4. The smart city perception image processing system equipped with a multimodal model according to claim 3 is characterized in that: The first three layers of the scene-specific feature extraction module are the same as the first three layers of the backbone network of the Yolov11 network structure. The scene-specific feature extraction module is embedded between the first three layers and the fourth layer of the backbone network of the Yolov11 network structure.

5. The smart city perception image processing system equipped with a multimodal model according to claim 3 is characterized in that: The size of the convolution kernel in the last layer of the Conv module structure is 1, and the step size of the convolution operation is 1. The last layer of the Conv module adjusts the dimension of the proprietary visual feature to be consistent with the second feature dimension output by the C3K2 module of the backbone network, outputs the first feature, and then uses the cat structure to fuse the first and second features.

6. The smart city perception image processing system equipped with a multimodal model according to claim 2, characterized in that: The visual group fusion and compression module includes a feature linking layer, a compression linear layer, a normalization function, and an activation layer. The feature linking layer is used to concatenate and fuse the original visual coding features, the enhanced visual coding features, and the proprietary visual coding features to obtain the first image feature. The compressed linear layer is used to perform feature compression and effective feature extraction on the first image feature to obtain effective features; The normalization function and activation layer are used to normalize and linearly transform the effective features respectively to obtain multi-scale image features.

7. The smart city perception image processing system equipped with a multimodal model according to claim 1, characterized in that: The multi-layer perception MLPs structure includes a multi-layer perceptron MLP module with multiple cycles and residual connections. The multi-layer perceptron MLP module includes a first fully connected layer, an activation function, and a second fully connected layer connected in sequence. The input multi-scale image features are subjected to multiple cycles and then residually connected with the output from the second fully connected layer to output the input image features.

8. A smart city perception image processing method equipped with a multimodal model, characterized in that: The following steps are involved: S1. Input the city perception image as the original inference image into the trained hybrid visual feature coding group for image feature coding. The hybrid visual feature coding group includes the original visual coding structure, the enhanced visual coding structure, and the proprietary visual coding structure, and outputs the original visual coding features, the enhanced visual coding features, and the proprietary visual coding features respectively. S2. Input the original visual coding features, enhanced visual coding features, and proprietary visual coding features into the visual group fusion and compression module for row fusion and compression, and output multi-scale image features; S3, input the multi-scale image features into the multi-layer perception MLPs structure for mapping to obtain input image features; S4, performing a text description on the city perception image, obtaining a corresponding text input instruction, inputting the text input instruction into a text instruction encoding structure, converting the text input instruction, and obtaining text features; S5. Input the input image features and text features into the trained large language model, concatenate the input image features and text features, and generate language response output.

9. The method for processing smart city perception images using a multimodal model according to claim 8, wherein: The proprietary visual encoding structure includes the Yolov11 network structure, a scene-specific feature extraction module, and a decision module. The training process of the proprietary visual encoding structure includes the following steps: S11. Construct a training data set, collect several perception pictures related to urban management and store them in the training data set, recorded as data1; S12, filter out the perception images corresponding to the proprietary scene from data1 and label these perception images with "yes" and save them in a proprietary image dataset, recorded as data2, label the perception images of the difference set of data1 and data2 with "no" and then combine the labeled data2 and the difference set into an image decision label dataset, recorded as data3; S13, the training is divided into auxiliary decision training, first stage training and second stage training. The auxiliary decision training is to use data3 to train the decision module to obtain the optimal binary target detection decision model and the optimal weight weight3 of the model; S14, the first stage of training is to use data1 to train the Yolov11 network structure to obtain the best target detection model and its weight weight1; S15, the second stage of training is to use data2 to train the scene-specific feature extraction module to obtain the optimal extraction model and its weight weight2; S16. According to weight1, weight2, and weight3, the weights of the Yolov11 network structure, the scene-specific feature extraction module, and the decision module are adjusted respectively to obtain a trained proprietary visual encoding structure.

10. The method for processing smart city perception images using a multimodal model according to claim 9, characterized in that: The decision module selects the resnet50 classification model.

Images