Smart city perception system based on visual coding group
Through visual coding group and dynamic weight allocation technology, the problem of the inability of visual feature fusion methods in existing technologies to adjust adaptively is solved, the accuracy of urban event judgment is improved and operation and maintenance costs are reduced.
Patent Information
- Application Number
- CN202510512366.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-23
- Publication Date
- 2025-09-19
AI Technical Summary
In existing technologies, the existing fusion method of multiple visual features cannot be adaptively adjusted, resulting in the inability to determine which dimension of visual features is more effective for perception results in different time and task scenarios, increasing operation and maintenance costs and reducing the accuracy of urban event judgment.
A smart city perception system based on visual coding groups is adopted. Dynamic weights are assigned to multiple visual coding features through the visual fusion and compression module. Combined with the multi-layer perception MLPs structure and large language model, the importance of visual features can be adaptively adjusted to match scene requirements.
It improves the accuracy of urban event perception, helps city managers better identify events in complex scenarios, and reduces operation and maintenance costs.
Smart Images
Figure CN120673219A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a smart city perception system based on a visual coding group. Background Art
[0002] In the smart city AI supervision system, event judgment based on accurate description of target attributes is crucial. This description can not only effectively distinguish the differences in the attributes of the same target, but also provide detailed event judgment for the smart city.
[0003] Existing smart city AI detection systems use a combination of multiple visual models to address smart city management issues. This can improve detection effectiveness to a certain extent, but it cannot fundamentally solve urban management issues. Instead, the complex logic increases operation and maintenance costs. Existing methods for fusing visual features from multiple dimensions use a simple concat fusion method, which does not consider the contribution of each dimension to the other. The importance of visual features from each dimension varies across different time and task scenarios, and the concat fusion method makes it impossible for managers to determine which dimension's visual features are most effective for perception. Furthermore, the importance of each dimension's visual features cannot be simply determined by manually setting the weight of each dimension's visual features. Therefore, an adaptive method is needed to dynamically adjust the importance of visual features from multiple dimensions to dynamically match the current scenario. Summary of the Invention
[0004] The purpose of the present invention is to provide a smart city perception system based on a visual coding group. In the visual fusion and compression module, multiple visual coding features are assigned corresponding weights to represent the scene requirements met by the original reasoning image, realize dynamic matching of scenes, and improve the accuracy of urban event perception.
[0005] To achieve the above object, the present invention proposes the following solutions:
[0006] In one aspect, the present invention provides a smart city perception system based on a visual coding group, comprising:
[0007] The visual coding group includes several visual coding structures with different functions, which are used to perform different visual coding processing on the same original reasoning image to obtain several corresponding visual coding features;
[0008] The visual fusion and compression module includes: a dynamic threshold module, which includes several dynamic threshold gating networks corresponding to each visual coding structure, each dynamic threshold gating network is used to adaptively assign a weight value to each visual coding feature;
[0009] A feature fusion module is used to fuse several visual coding features according to the weight value of each visual coding feature and output a fused feature;
[0010] Feature compression module, used to compress fused features and output multi-scale image features;
[0011] Multi-layer perception (MLPs) structure is used to map multi-scale image features to language semantic space to obtain input image features;
[0012] The large language model is used to perform event reasoning and judgment based on the input image features and the text features corresponding to the original inference image, and ultimately output the urban event perception results.
[0013] In some specific embodiments, the dynamic threshold gating network includes a multi-head attention structure, a compression structure, and a weight distribution structure connected in sequence, wherein the multi-head attention structure is used to perform multi-scale feature extraction on the input visual encoding features;
[0014] The compression structure is used to compress multi-scale features into image coding feature outputs of uniform dimension. The image coding feature output dimensions of the compression structure of each dynamic threshold gating network are consistent.
[0015] The weight distribution structure is used to calculate the proportion of the current input image coding feature in the sum of the image coding features output by the compression structure of all dynamic threshold gating networks, and obtain the weight value corresponding to the current input visual coding feature.
[0016] In some specific embodiments, the multi-head attention structure includes a first normalization layer, a second multi-head attention layer, a third normalization layer and a fourth feedforward neural network layer connected in sequence, wherein the output of the first normalization layer and the output of the second multi-head attention layer are input into the third normalization layer after residual connection, and the output of the third normalization layer and the output of the fourth feedforward neural network layer are used as the output of the multi-head attention structure after residual connection.
[0017] In some specific embodiments, the compression structure includes a first layer compression structure and a second layer compression structure connected in sequence, the first layer compression structure compresses the dimension of the multi-scale feature to 1 / 2 of the original, and the second layer compression structure further compresses the dimension of the multi-scale feature layer to 1.
[0018] In some specific embodiments, the first-layer compression structure and the second-layer compression structure both include a linear layer, a normalization layer, and an activation layer.
[0019] In some specific embodiments, the sum of the weight values output by the weight allocation structures of the dynamic threshold gating networks is 1.
[0020] In some embodiments, the visual encoding group includes:
[0021] The original visual coding structure is used to receive the original inference image for original visual coding and output the original visual coding features;
[0022] Enhanced visual coding structure, which is used to receive the original inference image, perform enhanced image coding, and output enhanced visual coding features;
[0023] The proprietary visual coding structure is used to receive the original inference image, perform proprietary visual image coding and conversion, and output proprietary visual coding features that are consistent with the original visual coding features and enhanced visual coding feature structure dimensions.
[0024] In some specific embodiments, the proprietary visual encoding structure includes: a Yolov11 network structure, a scene-specific feature extraction module and a decision module, wherein:
[0025] The decision module is used to perform a binary classification decision on the received raw reasoning image to determine whether further scene-specific features need to be extracted. If the judgment result is yes, the raw reasoning image is input into the scene-specific feature extraction module and the Yolov11 network structure respectively. If the judgment result is no, the raw reasoning image is directly input into the Yolov11 network structure;
[0026] The scene-specific feature extraction module is used to extract the specific features of the original inference image and input the first feature into the backbone network of the Yolov11 network structure;
[0027] The Yolov11 network structure includes a backbone network, a neck network and a head network. The backbone network is embedded with a scene-specific feature extraction module, and decides whether to fuse it with the first feature output by the scene-specific feature extraction module based on the judgment result of the decision module.
[0028] In some specific embodiments, the four-layer structure of the scene-specific feature extraction module is: two layers of identical Conv modules connected in series, one layer of C3K2 modules, and a last layer of Conv modules, wherein the two layers of identical Conv modules connected in series and one layer of C3K2 modules are used to extract proprietary visual features from the original reasoning image, and the last layer of Conv module is used to match the proprietary visual features with the feature dimensions of the Yolov11 network structure and output the first feature.
[0029] In some specific implementation schemes, the first three layers of the scene-specific feature extraction module are the same as the first three layers of the backbone network of the Yolov11 network structure. The scene-specific feature extraction module is embedded between the first three layers and the fourth layer of the backbone network of the Yolov11 network structure. The size of the convolution kernel in the last layer of the Conv module structure is 1, and the step size of the convolution operation is 1. The last layer of the Conv module adjusts the dimension of the proprietary visual feature to be consistent with the second feature dimension output by the C3K2 module of the backbone network, and then outputs the first feature, and then uses the cat structure to fuse the first feature and the second feature.
[0030] The present invention has the beneficial effects:
[0031] In order to match the scene requirements corresponding to the original reasoning picture, the present invention assigns different weights to the multiple visual coding features output by the multi-visual visual coding group. Adaptive adjustment of the weight of each visual coding feature can improve the accuracy of the model, thereby helping city managers better perceive real events happening in the city. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 A schematic diagram of the composition of a smart city perception system based on a visual coding group provided by an embodiment of the present invention;
[0033] Figure 2 A schematic diagram of the composition of the visual fusion and compression module provided in an embodiment of the present invention;
[0034] Figure 3 A schematic diagram of the composition of a dynamic threshold module provided in an embodiment of the present invention;
[0035] Figure 4 A schematic diagram of the multi-head attention structure provided by an embodiment of the present invention;
[0036] Figure 5 A diagram showing the system composition of the LLaVA multimodal large model provided by an embodiment of the present invention;
[0037] Figure 6 A schematic diagram of the composition of a proprietary visual coding structure provided by an embodiment of the present invention;
[0038] Figure 7 A schematic diagram of the proprietary visual feature extraction training process provided by an embodiment of the present invention. Figure 8 Schematic diagram of the multi-layer perception MLPs structure provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0039] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0040] Unless otherwise specifically stated, the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention.
[0041] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0042] Additionally, descriptions of well-known structures, functions, and configurations may be omitted for clarity and conciseness. Those skilled in the art will recognize that various changes and modifications can be made to the examples described herein without departing from the spirit and scope of the present disclosure.
[0043] Technologies, methods and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods and equipment should be considered part of the authorization specification.
[0044] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0045] Example 1
[0046] like Figure 1 As shown, this embodiment provides a smart city perception system based on a visual coding group, including:
[0047] The visual coding group includes several visual coding structures with different functions, which are used to perform different visual coding processing on the same original reasoning image to obtain several corresponding visual coding features;
[0048] The visual fusion and compression module includes: a number of dynamic threshold gating networks corresponding to each visual coding structure, each dynamic threshold gating network is used to adaptively assign weight values to each visual coding feature;
[0049] A feature fusion module is used to fuse several visual coding features according to the weight value of each visual coding feature and output a fused feature;
[0050] Feature compression module, used to compress fused features and output multi-scale image features;
[0051] Multi-layer perception (MLPs) structure is used to map multi-scale image features to language semantic space to obtain input image features;
[0052] The large language model is used to perform event reasoning and judgment based on the input image features and the text features corresponding to the original inference image, and ultimately output the urban event perception results.
[0053] Specifically, the original inference images referred to in this application refer to various events occurring in a city, as detected by cameras or city surveillance equipment mounted on mobile vehicles (e.g., city buses). For example, these vehicles can capture images of peddlers on city roads. In this context, peddlers are vehicles that stop on various city roads to sell goods, such as fruits, vegetables, and snacks. They can use four-wheeled trucks, three-wheeled trucks, or other cargo-carrying vehicles like three-wheeled bicycles. Their characteristic characteristic is that these vehicles, carrying goods for sale, remain in a certain location for a long period of time. In the context of peddlers, accurately detecting whether peddlers are occupying the road at their location through images would help city managers manage peddlers and prevent them from occupying the road. However, due to the complexity of peddlers, misidentification is prone to occur. For example, a vehicle carrying goods but not selling goods while in motion, simply transporting goods, could be mistakenly identified as a peddler. Alternatively, vehicles such as fruit carts temporarily parked on the road could be mistakenly identified as peddlers. In view of this situation, the prior art uses target detection to identify whether the person is a traveling peddler, but it can only provide the target category, and complex scenes cannot be distinguished or even provide wrong judgments. For this reason, it is very necessary to adopt a large model with text description and reasoning ability that takes into account scene factors or an attribute text description. Therefore, this application uses a multimodal large model to solve these problems. In addition, in order to improve the recognition accuracy, this application also proposes to improve the existing Yolov11 network structure and add a scene-specific feature extraction module. It can optimize the problem of inaccurate processing of pictures of mobile peddlers in specific scenes, such as scenes obtained in bad weather, foggy days or strong light, and improve the model's processing accuracy for pictures taken in these specific scenes. And the multimodal large model adopts the LLaVA multimodal large model.
[0054] The following is an explanation of each of the above modules:
[0055] 1. Visual encoding group
[0056] The visual coding group includes: an original visual coding structure, which is used to receive the original reasoning picture for original visual coding and output the original visual coding features; an enhanced visual coding structure, which is used to receive the original reasoning picture for enhanced image coding and output enhanced visual coding features; a proprietary visual coding structure, which is used to receive the original reasoning picture for proprietary visual image coding and convert it, and output proprietary visual coding features that are consistent with the dimensions of the original visual coding features and enhanced visual coding feature structures.
[0057] 1.1. Enhanced visual encoding module or original visual encoding structure
[0058] The enhanced visual coding module or the original visual coding structure basically uses a VIT structure model, which in turn uses a transformer-based structure. For example, the enhanced visual coding structure uses the SAM2 model. The existing transformer structure proposes an encoder-decoder structure, while the underlying structure uses a transformer and FFN structure. As can be seen from the figure, the transformer structure consists of Q, K, and V, where Q, K, and V are implemented using conventional linear convolution. The transformer structure is finally implemented using the following formula 1, which is calculated as follows:
[0059]
[0060] where d k Represents the feature expression dimension.
[0061] The FFN structure consists of two linear convolutions and a ReLU activation function, and its calculation formula is as follows:
[0062] FFN(x)=max(0,xW1+b1)w2+b2 (2)
[0063] 1.2 Proprietary Visual Encoding Structure
[0064] like Figure 6 As shown in the figure, the proprietary visual encoding structure is an improvement on the Yolov11 network structure. It is used to solve conditions such as lighting and defogging. It is a method for obtaining proprietary visual features for urban management. This is a feature information expression that the multimodal and enhancement modules do not have. Specifically, it includes:
[0065] 1.2.1 Decision Module
[0066] The decision module is used to perform a binary classification decision on the received original reasoning image to determine whether further scene-specific features need to be extracted. If the judgment result is yes, the original reasoning image is input into the scene-specific feature extraction module and the Yolov11 network structure respectively. If the judgment result is no, the original reasoning image is directly input into the Yolov11 network structure. The decision module uses the resnet50 classification model as the decision model. 1.2.2 Scene-specific feature extraction module
[0067] The scene-specific feature extraction module is used to extract the specific features of the original inference image and input the first feature into the backbone network of the Yolov11 network structure; Figure 6 As shown, the four-layer structure of the scene-specific feature extraction module is: two identical layers of tandem Conv modules, a layer of C3K2 modules, and a final Conv module. The two identical layers of tandem Conv modules and the C3K2 module are used to extract specific visual features from the original inference image. The final Conv module is used to match the specific visual features with the feature dimensions of the Yolov11 network structure and output the first feature. The first three layers of the scene-specific feature extraction module are identical to the first three layers of the Yolov11 network backbone network. The scene-specific feature extraction module is embedded between the first three layers of the Yolov11 backbone network and the fourth layer.
[0068] In order to enhance the feature adaptability structure, in the last layer Conv module structure, Conv k=1, s=1 is the adjusted feature, the size of the convolution kernel is 1, and the step size of the convolution operation is 1. The last layer Conv module adjusts the dimension of the proprietary visual feature to be consistent with the second feature dimension output by the C3K2 module of the backbone network, outputs the first feature (if the decision module determines that the scene-specific feature extraction module needs to be used, the feature is output), and then uses the cat structure to directly add the first and second features for fusion.
[0069] 1.2.3 Yolov11 Network Structure
[0070] The Yolov11 network structure in this embodiment is the same as the existing structure, except that a scene-specific feature extraction module is embedded between the first three layers and the fourth layer to accurately identify images in specific scenes, such as Figure 6 As shown in the figure, the Yolov11 network structure includes a backbone network (Backbone), a neck network (Neck) and a head network (Head), wherein the backbone network is embedded with a scene-specific feature extraction module, and decides whether to fuse the first feature output by the scene-specific feature extraction module based on the judgment result of the decision module.
[0071] 2. Visual Fusion and Compression Module
[0072] like Figure 2 As shown, for the visual coding features output by the three visual coding structures of the visual coding group, three dynamic threshold gating networks are set up accordingly. The weight value of each visual coding feature is determined, and the residual of each visual coding feature and its corresponding weight value is calculated and input into the feature fusion module. Then, the feature fusion module connects the feature values output by the three dynamic threshold gating networks and inputs them into the feature compression module for compression and output. Specifically:
[0073] 2.1 Dynamic Threshold Module
[0074] like Figure 3 As shown, the dynamic threshold module includes three dynamic threshold gating networks: a first dynamic threshold gating network for determining the weight value of the original visual coding feature, a second dynamic threshold gating network for determining the weight value of the enhanced visual coding feature, and a third dynamic threshold gating network for determining the weight value of the proprietary visual coding feature. Each dynamic threshold gating network includes a multi-head attention structure, a compression structure, and a weight distribution structure connected in sequence. Specifically:
[0075] 2.1.1 Multi-head Attention Block
[0076] The multi-head attention structure is used to extract multi-scale features from the input visual encoding features; Figure 4 As shown, the multi-head attention structure includes a first normalization layer LayerNorm, a second multi-head attention layer Mult-HeadAttention, a third normalization layer LayerNorm and a fourth feedforward neural network layer FFN connected in sequence, wherein the output of the first normalization layer and the output of the second multi-head attention layer are input into the third normalization layer after residual connection, and the output of the third normalization layer and the output of the fourth feedforward neural network layer are used as the output of the multi-head attention structure after residual connection.
[0077] Since the format of each visual coding feature is the same, (BLC), B represents the batch of input images, L represents token, and C represents each token expressed as a vector. Before inputting the multi-head attention structure, each dynamic threshold gating network needs to input the visual coding feature and perform corresponding position encoding on the input feature to obtain a feature format that conforms to the multi-head attention structure input. For example, the original visual coding feature F oAfter (B, L, C) is input into the multi-head attention structure of the first dynamic threshold gating network, multi-scale features (B, L, C) are obtained to enhance the visual encoding feature F e After (B, L, C) is input into the multi-head attention structure of the first dynamic threshold gating network, multi-scale features (B, L, C) are obtained, and the proprietary visual encoding feature F p After (B, L, C) is input into the multi-head attention structure of the first dynamic threshold gating network, multi-scale features (B, L, C) are obtained.
[0078] 2.1.2 Compression Structure
[0079] The compression structure is used to compress multi-scale features into image coding feature outputs of uniform dimension. The image coding feature output dimensions of the compression structure of each dynamic threshold gating network are consistent.
[0080] Specifically, the compression structure includes a first and second compression layer connected in sequence. Both the first and second compression layers include a linear layer (Linear), a normalization layer (LN), and an activation layer (RELU). The first compression layer compresses the dimensionality of the multi-scale feature layer to 1 / 2 of its original size, and the second compression layer further compresses the dimensionality of the multi-scale feature layer to 1.
[0081] It can be seen that after the multi-scale feature (B, L, C) passes through the first layer of compression structure, the feature dimension becomes: (B, L, C / / 2), and after passing through the second layer of compression structure again, the feature dimension becomes: (B, L, 1). The compression modules of the three dynamic threshold gating networks output F o1 (B, L, 1), F e1 (B, L, 1), F p1 (B, L, 1).
[0082] 2.1.3 Weight Distribution Structure (Softmax Function)
[0083] The weight distribution structure is used to calculate the proportion of the current input image coding feature in the sum of the image coding features output by the compression structure of all dynamic threshold gating networks, and obtain the weight value corresponding to the current input visual coding feature.
[0084] For example, the weight value output by the first dynamic threshold gating network is μ, the weight value output by the second dynamic threshold gating network is ρ, and the weight value output by the third dynamic threshold gating network is τ, where μ+ρ+τ=1. The weight value calculation process corresponding to each visual coding feature is:
[0085] The weight value of the original visual encoding feature:
[0086] Enhanced visual encoding features:
[0087] Proprietary visual encoding features:
[0088] 2.2 Feature Fusion Module
[0089] The feature fusion module is used to fuse several visual coding features according to the weight value of each visual coding feature and output the fusion feature F = Concat(μF o， ρF e， τF p) ;
[0090] 2.3 Feature Compression Module
[0091] The feature compression module consists of a compressed linear layer (nn.linear), a normalization function (LN), and an activation function (RELU). The compressed linear layer compresses the fused features and extracts valid features, resulting in effective features. The normalization function and activation layer, respectively, normalize and linearly transform the effective features to obtain multi-scale image features. Feature compression using the nn.liear method reduces high-dimensional features to low dimensions through linear mapping, removing redundant information, focusing on key features, and meeting the original LLaVA model feature format. For example, the 768-dimensional features output by ViT are compressed to 50 dimensions. Normalization and activation functions are then applied. Normalization normalizes the distribution of input data at each layer, accelerating training, stabilizing the neural network, and improving model performance. The activation function introduces nonlinear transformations, enabling neural networks to learn and express complex patterns.
[0092] Specifically, the original visual coding features are obtained from the original visual coding structure, and their format is (BLC); the enhanced visual coding features are obtained from the enhanced visual coding structure, and their format is (BL1C1); the proprietary visual coding features are obtained from the proprietary visual coding structure, and their format is (BC3H3W3). In order to make the proprietary visual coding features consistent with the formats of the original visual coding features and the enhanced visual coding features, an adaptive visual feature conversion structure is provided between the hybrid visual feature coding group and the visual group fusion and compression module, which converts the formats of the proprietary visual coding features, the original visual coding features, and the enhanced visual coding features into the same format. The adaptive visual feature conversion structure includes three MLP feature conversion modules, which process the proprietary visual coding features, the original visual coding features, and the enhanced visual coding features respectively. Among them, the proprietary visual coding features are converted by first converting (BC3H3W3) into (BC3H3*W3) through format conversion, and then performing position conversion (BH3*W3C3) to finally obtain (BL3C3). Where L3=H3*W3.
[0093] As can be seen, the output format of the original visual coding structure, the enhanced visual coding structure, and the proprietary visual coding structure are all [BLC], where B represents the input image batch, L represents the token, and C represents the dimensional representation of each token. However, the original LLAVA model image input requires BLC structure data. To adapt to different scene requirements and dynamically match scenes, the visual fusion and compression module in this application dynamically assigns different weights to each visual coding feature to more effectively integrate the fused feature representation.
[0094] 3. Multi-layer perception MLPs structure
[0095] like Figure 8 As shown in the figure, the multi-layer perception MLPs structure is obtained by multiple cycles of a multi-layer perceptron MLP module, and an MLP structure with residual connection is added, which specifically includes two fully connected layers and an activation function, as shown in the formula below.
[0096] M(x)=x+[W2(σ(W1*x+b1))+b2]
[0097] Among them, σ(·) is the activation function, and this application chooses RELU as the activation function; W1 and W2 represent nn.Linear linear weights, respectively, and b1 and b2 are the corresponding biases. A multi-layer perception MLPs structure is adopted, so that the input multi-scale image features are passed through multiple cycles Nx and then connected with the output of the second fully connected layer through a residual connection to output the input image features that meet the input of the large language model. Since the spatial state and feature dimension of the multi-scale image features are higher than those of the text features, in order to enable the multi-scale image features and text features to be spliced in the large language model, it is necessary to map the multi-scale image features to the language semantic space so that they are consistent with the state of the text features.
[0098] 4. Large Language Model
[0099] like Figure 5 As shown, in this embodiment, the entire image processing system is improved based on the LLaVA multimodal large model. The LLaVA multimodal large model adopts the CLIP visual feature network as the visual encoder. The multi-scale image feature XvImage after fusion and compression by the visual group fusion and compression module is converted into a language embedding token Hv in the word embedding space after passing through the multi-layer perception MLPs structure. The text input instruction Xq language instruction is converted into a text feature Hq through the text instruction encoding structure. Hv and Hq are passed through the original large language model Language Model of the LLaVA multimodal large model to generate a language response Language Response Xa output.
[0100] In order to obtain the modules mentioned above, it is necessary to collect a large number of pictures of urban mobile vendors to train various scenes, such as Figure 7 As shown, for example, the training process for the mobile vendor scenario includes the following steps:
[0101] Step 1: Proprietary visual feature extraction training
[0102] S11. Construct a training dataset. Use the vehicle-mounted device to collect several pictures of mobile vendors related to urban management and store them in the training dataset, which is recorded as data1.
[0103] S12. Filter out mobile vendor scene images corresponding to proprietary scenarios (proprietary scenarios include bad weather, foggy days, or strong light scenes) from data1 and label these mobile vendor scene images with "yes" and save them in a proprietary image dataset, recorded as data2. Label the mobile vendor scene images in the difference set of data1 and data2 with "no" and then combine the labeled data2 and the difference set into an image decision label dataset, recorded as data3.
[0104] S13, the training is divided into auxiliary decision training, first-stage training and second-stage training. The auxiliary decision training uses data3 to train the decision module. The decision module selects the resnet50 classification model to obtain the optimal binary target detection decision model and the optimal weight weight3 of the model. The optimal binary target detection decision model is recorded as model3.
[0105] S14, the first stage of training is to use data1 to train the Yolov11 network structure to obtain the best target detection model and its weight weight1, and the best target detection model is recorded as model1;
[0106] S15. Use the same module as Yolov11 to build a proprietary feature extraction module. This module has two Convs and one C3K2 for special processing of proprietary scenes. The last Conv is used to extract proprietary scene features that adapt to the original model feature space of Yolov11. Finally, use the cat structure to directly add them together for fusion. The second stage of training is to use data2 to train the scene-specific feature extraction module. At the same time, use weight1 to assign values to the Yolov11 network structure during training. When training the scene-specific feature extraction module, freeze all structures of the Yolov11 network structure and only train the scene-specific feature extraction module to obtain the optimal extraction model, recorded as model2, and its corresponding optimal weight weight2;
[0107] S16. According to weight1, weight2, and weight3, the weights of the Yolov11 network structure, the scene-specific feature extraction module, and the decision module are adjusted respectively to obtain a trained proprietary visual encoding structure.
[0108] Step 2: Proprietary Visual Feature Extraction, Embedding, and Inference
[0109] The optimal models corresponding to the Yolov11 network structure, scene-specific feature extraction module, and decision module obtained above are embedded in the system. The specific process is as follows:
[0110] S21. Build a proprietary visual encoding structure. The integrated decision module + scene-specific feature extraction module is embedded in the Yolov11 network structure, such as Figure 6 As shown in the figure, the weights trained above are assigned to each structure to obtain a complete vision-specific feature extraction network. The original trained head module has been removed here, and the features are directly output in the format of (B C3 H3 W3).
[0111] S22, binary classification decision model reasoning. Input the reasoning image I and perform a binary classification decision to determine whether further extraction of unique visual features is required. If yes, the decision is yes; otherwise, the decision is no.
[0112] S23: Dedicated visual feature reasoning. If the decision in step S22 is yes, the inference image I is input into the scene-specific feature extraction module for reasoning, and the final output is the first feature F1; if the decision in step S22 is no, no reasoning is performed.
[0113] S24: Inference of proprietary visual feature embedding. Image I is processed through the first three layers of Yolov11 (Conv-Conv and C3K2) to obtain the second feature F2. If the decision in step S22 is yes, features F1 and F2 are directly added together to form the fused feature F3. If the decision in step S22 is no, feature F3 is equal to feature F2.
[0114] S25, model inference. After obtaining F3 in step S24, continue to complete the processing of the remaining structure of Yolov11, and finally obtain the proprietary visual encoding feature output, whose format is (B C3 H3 W3)
[0115] Step 3: Training of multimodal large models
[0116] S31. Construct a dataset data4. The data in this dataset corresponds to the multimodal sample data annotation of urban management. It is a text annotation of an image and multiple rounds of dialogue.
[0117] S32. Train the LLaVA multimodal model. Input data4 into the LLaVA multimodal model for model training to obtain the optimal model. While the LLaVA multimodal model is training, the structure in the visual encoding group needs to be frozen, and no weight update is performed.
[0118] S33. Model deployment. The proprietary visual encoding structure obtained in step 1 and the optimal model obtained in step S32 are deployed in a cloud server and applied to smart city-related tasks.
[0119] It is understandable that this application constructs a visual feature coding group based on a multimodal large model. In addition to retaining the original visual feature extraction, it adds additional enhanced visual coding to further improve the generalization ability of its visual features. In addition, a proprietary visual feature extraction module is provided for urban management scenarios. This module is pre-trained in existing scenes to obtain proprietary features adapted to the scene. Since the existing visual fusion and compression modules are generally fused through simple concat, which is relatively rough, it is impossible to determine which visual coding feature is more effective for the scene task. Different scenes may require more original visual features or more enhanced visual features at certain times. It is also impossible to manually set the weight value of each visual coding feature to make it meet the scene requirements. In order to solve this problem, this embodiment proposes a system that adaptively adjusts the weight of each feature coding feature to dynamically match the scene, and adjusts these parameters according to the task or scene requirements.
[0120] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Based on the technical essence of the present invention and within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement of the above embodiment shall still fall within the scope of protection of the technical solution of the present invention.
Claims
1. A smart city perception system based on visual coding groups, characterized by: include: The visual coding group includes several visual coding structures with different functions, which are used to perform different visual coding processing on the same original reasoning image to obtain several corresponding visual coding features; The visual fusion and compression module includes: a dynamic threshold module, which includes several dynamic threshold gating networks corresponding to each visual coding structure, each dynamic threshold gating network is used to adaptively assign a weight value to each visual coding feature; A feature fusion module is used to fuse several visual coding features according to the weight value of each visual coding feature and output a fused feature; Feature compression module, used to compress fused features and output multi-scale image features; Multi-layer perception (MLPs) structure is used to map multi-scale image features to language semantic space to obtain input image features; The large language model is used to perform event reasoning and judgment based on the input image features and the text features corresponding to the original inference image, and ultimately output the urban event perception results.
2. The smart city perception system based on visual coding groups according to claim 1 is characterized in that: The dynamic threshold gating network includes a multi-head attention structure, a compression structure, and a weight distribution structure connected in sequence. The multi-head attention structure is used to extract multi-scale features from the input visual encoding features. The compression structure is used to compress multi-scale features into image coding feature outputs of uniform dimension. The image coding feature output dimensions of the compression structure of each dynamic threshold gating network are consistent. The weight distribution structure is used to calculate the proportion of the current input image coding feature in the sum of the image coding features output by the compression structure of all dynamic threshold gating networks, and obtain the weight value corresponding to the current input visual coding feature.
3. The smart city perception system based on visual coding groups according to claim 2 is characterized in that: The multi-head attention structure includes a first normalization layer, a second multi-head attention layer, a third normalization layer and a fourth feedforward neural network layer connected in sequence, wherein the output of the first normalization layer and the output of the second multi-head attention layer are input into the third normalization layer after residual connection, and the output of the third normalization layer and the output of the fourth feedforward neural network layer are used as the output of the multi-head attention structure after residual connection.
4. The smart city perception system based on visual coding groups according to claim 2 is characterized in that: The compression structure includes a first-layer compression structure and a second-layer compression structure connected in sequence. The first-layer compression structure compresses the dimension of the multi-scale feature to 1 / 2 of the original, and the second-layer compression structure further compresses the dimension of the multi-scale feature layer to 1.
5. The smart city perception system based on visual coding groups according to claim 2 is characterized in that: Both the first-layer compression structure and the second-layer compression structure include a linear layer, a normalization layer, and an activation layer.
6. The smart city perception system based on visual coding groups according to claim 2 is characterized in that: The sum of the weight values output by the weight distribution structure of each dynamic threshold gating network is 1.
7. The smart city perception system based on visual coding groups according to claim 1 is characterized in that: The visual encoding group includes: The original visual coding structure is used to receive the original inference image for original visual coding and output the original visual coding features; Enhanced visual coding structure, which is used to receive the original inference image, perform enhanced image coding, and output enhanced visual coding features; The proprietary visual coding structure is used to receive the original inference image, perform proprietary visual image coding and conversion, and output proprietary visual coding features that are consistent with the original visual coding features and enhanced visual coding feature structure dimensions.
8. The smart city perception system based on visual coding groups according to claim 7 is characterized in that: The proprietary visual encoding structure includes: Yolov11 network structure, scene-specific feature extraction module and decision module, among which, The decision module is used to perform a binary classification decision on the received raw reasoning image to determine whether further scene-specific features need to be extracted. If the judgment result is yes, the raw reasoning image is input into the scene-specific feature extraction module and the Yolov11 network structure respectively. If the judgment result is no, the raw reasoning image is directly input into the Yolov11 network structure; The scene-specific feature extraction module is used to extract the specific features of the original inference image and input the first feature into the backbone network of the Yolov11 network structure; The Yolov11 network structure includes a backbone network, a neck network and a head network. The backbone network is embedded with a scene-specific feature extraction module, and decides whether to fuse it with the first feature output by the scene-specific feature extraction module based on the judgment result of the decision module.
9. The smart city perception system based on visual coding groups according to claim 8 is characterized in that: The four-layer structure of the scene-specific feature extraction module is: two identical layers of Conv modules connected in series, one layer of C3K2 module, and the last layer of Conv module. The two identical layers of Conv modules connected in series and one layer of C3K2 module are used to extract proprietary visual features from the original inference image. The last layer of Conv module is used to match the proprietary visual features with the feature dimensions of the Yolov11 network structure and output the first feature.
10. The smart city perception system based on visual coding groups according to claim 9 is characterized in that: The first three layers of the scene-specific feature extraction module are the same as the first three layers of the backbone network of the Yolov11 network structure. The scene-specific feature extraction module is embedded between the first three layers and the fourth layer of the backbone network of the Yolov11 network structure. The size of the convolution kernel in the last layer of the Conv module structure is 1, and the step size of the convolution operation is 1. The last layer of the Conv module adjusts the dimension of the proprietary visual feature to be consistent with the second feature dimension output by the C3K2 module of the backbone network, and then outputs the first feature, and then uses the cat structure to fuse the first feature and the second feature.