Multi-modal large model based on dynamic decision hybrid expert mapping

Abstract

The invention discloses a multi-modal large model based on dynamic decision hybrid expert mapping, which comprises a visual coding group comprising a plurality of visual coding structures with different functions, a visual fusion and compression module, a dynamic decision hybrid expert mapping module and a large language model, the dynamic decision hybrid expert mapping module comprises a hybrid expert module and a shared expert module which respectively process the multi-scale image features to obtain high-discrimination features and comprehensive features, and a decision expert module which is used for carrying out importance estimation on the multi-scale image features; the decision connection module and the self-adaptive projection mapping layer are used for connecting the high-discrimination features and the comprehensive features according to importance estimated values, and the dynamic decision hybrid expert mapping module intelligently distributes parameters to different expert networks and dynamically adjusts weight configuration of experts. And the internal structure of the model can be automatically optimized according to different inputs, so that higher processing precision is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a multimodal large model based on dynamic decision hybrid expert mapping. Background Art

[0002] Existing smart city AI detection systems use a combination of multiple visual models to address smart city management issues. This improves detection effectiveness to a certain extent, but it cannot fundamentally solve urban management issues. Instead, the complex logic increases operational costs. For the multiple visual features output by the combined visual models, concat fusion or adaptive fusion strategies can be used. Adaptive fusion strategies enable efficient integration of multiple visual features, resulting in highly task-relevant visual feature representations. However, to further enhance these features and enable their more effective embedding into large language models, achieving cross-modal understanding and generation capabilities is crucial.

[0003] The main goal of embedding visual features into a large multimodal model architecture is to promote a deep understanding of the relationship between text and images, thereby achieving significant results in complex tasks such as image captioning, visual question answering, and text-guided image generation. Considering that the model needs to accurately handle tasks involving multiple modalities such as text and images, it is particularly important to have a powerful and complex structure. For example, the Q-former structure demonstrates its importance in this scenario. Through a more sophisticated and efficient cross-modal fusion strategy, it enhances the model's ability to process multimodal data, improves accuracy and depth of understanding, and thus significantly improves the performance of multimodal tasks.

[0004] However, the processing accuracy of this structure is not high. Therefore, it is necessary to improve the multimodal large model to improve the processing accuracy of the model. Summary of the Invention

[0005] The present invention aims to provide a large multimodal model based on dynamic decision-making hybrid expert mapping. By combining the hybrid expert structure with a dynamic mechanism allocation strategy, a dynamic decision-making hybrid expert mapping module is constructed. This module adaptively assigns parameters to different expert modules and dynamically adjusts the weights of each expert, enabling the model to automatically optimize its internal structure based on input variations, thereby achieving higher processing accuracy.

[0006] To achieve the above object, the present invention proposes the following solutions:

[0007] In one aspect, the present invention provides a multimodal large model based on dynamic decision hybrid expert mapping, comprising:

[0008] The visual coding group includes several visual coding structures with different functions, which are used to perform different visual coding processing on the same original reasoning image to obtain several corresponding visual coding features;

[0009] The visual fusion and compression module is used to fuse and compress several visual coding features and output multi-scale image features;

[0010] Dynamic decision-making hybrid expert mapping module, including:

[0011] The hybrid expert module is used to select several expert modules from multiple expert modules according to the dynamic routing mechanism to process the dimensional features of different dimensions in the multi-scale image features respectively, and connect the features processed by several expert modules to obtain highly discriminative features;

[0012] The shared expert module is used to process the dimensional features of different dimensions in the multi-scale image features separately and output the comprehensive features that integrate multimodal information;

[0013] Decision expert module, used to estimate the importance of each dimension of multi-scale image features;

[0014] The decision connection module is used to obtain the connection weights corresponding to the high-discriminative features and the comprehensive features based on the importance of the features of each dimension, and to connect the high-discriminative features and the comprehensive features according to their respective connection weights to obtain the decision features;

[0015] Adaptive projection mapping layer, used to map decision features to language semantic space to obtain input image features;

[0016] The large language model is used to perform event reasoning and judgment based on the input image features and the text features corresponding to the original inference image, and ultimately generate language response output.

[0017] In some specific implementations, the decision expert module includes: a layer of decision experts using an MLP structure, which is used to estimate the importance of each dimensional feature of the multi-scale image feature to obtain a predicted value in the format of (BL 2C); where B represents the original inference image batch, L represents the dimensional feature token, and C represents the dimensional expression of each token;

[0018] A shape change module is used to transform the predicted estimate to obtain a transformed estimate in the format of (BLC 2);

[0019] The activation function is used to process the conversion estimate to obtain the importance estimate of each visual feature.

[0020] In some specific embodiments, the hybrid expert module includes: a routing gating network for determining the expert modules involved in dimensional feature processing and matching the dimensional features of each dimension of the multi-scale image feature with the multiple expert modules;

[0021] Multiple expert modules, each expert module processes the dimensional features assigned from the routing gating network as an independent sub-network;

[0022] The connection module is used to aggregate the processed dimensional features output by each expert module into a unified feature expression to obtain highly discriminative features.

[0023] In some specific implementation schemes, the high-discriminative feature y output by the connection module is expressed as follows:

[0024]

[0025] Where i represents the i-th expert module, n represents the number of n expert modules; E i (x) represents the dimension feature token value assigned by the routing gating network selected by the i-th expert, and G(x) represents the normalized weight value output after each dimension feature token value x is normalized by n experts.

[0026] In some specific implementation schemes, the routing gating network adopts a noise-added gating network method. First, noise and linear convolution are added to each dimension feature token value x to obtain the first weight H(x) of the n expert modules assigned to each dimension feature token value. The expression is as follows:

[0027] H(x)=W g x+StandardNormal()·Softplus(W noise x)

[0028] Among them, StandardNormal() represents the standard normal distribution, W g With W noise Both represent linear convolution weights.

[0029] Select k expert modules from n expert modules. For the selected k expert modules, the output value of the i-th expert module is retained. Use the Softmax function and the first weight H(x) to normalize the n experts corresponding to each dimension feature token value x, so that the output values ​​of the unselected nk experts become 0, and obtain the normalized weight value G(x):

[0030] G(x)=Softmax(TopK(H(x),k)).

[0031] In some specific implementation schemes, the shared expert module adopts a 1-layer MLP structure.

[0032] In some specific embodiments, the adaptive projection mapping layer adopts a multi-layer perception MLPs structure, which includes a multi-layer perception MLP module with multiple cycles and residual connections. The multi-layer perception MLP module includes a first fully connected layer, an activation function, and a second fully connected layer connected in sequence. The input decision features are subjected to multiple cycles and then residually connected with the output from the second fully connected layer to output the input image features.

[0033] In some specific embodiments, the visual coding group includes an original visual coding structure, an enhanced visual coding structure, and a proprietary visual coding structure, wherein the proprietary visual coding structure includes: a Yolov11 network structure, a scene-specific feature extraction module, and a decision module, wherein:

[0034] The decision module is used to perform a binary classification decision on the received raw reasoning image to determine whether further scene-specific features need to be extracted. If the judgment result is yes, the raw reasoning image is input into the scene-specific feature extraction module and the Yolov11 network structure respectively. If the judgment result is no, the raw reasoning image is directly input into the Yolov11 network structure;

[0035] The scene-specific feature extraction module is used to extract the specific features of the original inference image and input the first feature into the backbone network of the Yolov11 network structure;

[0036] The Yolov11 network structure includes a backbone network, a neck network and a head network. The backbone network is embedded with a scene-specific feature extraction module, and decides whether to fuse it with the first feature output by the scene-specific feature extraction module based on the judgment result of the decision module.

[0037] In some specific implementation schemes, the structure of the scene-specific feature extraction module is the same as the first four layers of the backbone network of the Yolov11 network structure, and the scene-specific feature extraction module is embedded between the first three layers and the fourth layer of the backbone network of the Yolov11 network structure.

[0038] In some specific embodiments, the visual fusion and compression module includes: a dynamic threshold module, the dynamic threshold module including a plurality of dynamic threshold gating networks corresponding to each visual coding structure, each dynamic threshold gating network being used to adaptively assign a weight value to each visual coding feature;

[0039] A feature fusion module is used to fuse several visual coding features according to the weight value of each visual coding feature and output a fused feature;

[0040] The feature compression module is used to compress the fused features and output multi-scale image features.

[0041] The present invention has the beneficial effects:

[0042] To address the needs of vertical urban management and oversight, this paper constructs a Dynamic Decision and Hybrid Expert Mapping Module (DD-HEMM). DD-HEMM combines a Mixed of Experts (MOE) architecture with a dynamic mechanism allocation strategy. Specifically, by intelligently assigning parameters to different expert networks and dynamically adjusting the weights of each expert, the model automatically optimizes its internal structure based on input, achieving higher processing accuracy.

[0043] In addition, in order to match the scene requirements corresponding to the original inference image, different weights are assigned to the multiple visual coding features output by the multi-visual visual coding group. Adaptive adjustment of the weights of each visual coding feature can improve the accuracy of the model, thereby helping city managers better perceive real events happening in the city. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the composition of a multimodal large model based on dynamic decision-making hybrid expert mapping provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of the composition of a dynamic decision-making hybrid expert mapping module provided in an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of the composition of the visual fusion and compression module provided in an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of the composition of a dynamic threshold module provided in an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of the multi-head attention structure provided by an embodiment of the present invention;

[0049] Figure 6 A schematic diagram of the composition of a proprietary visual coding structure provided by an embodiment of the present invention;

[0050] Figure 7 A schematic diagram of the proprietary visual feature extraction training process provided by an embodiment of the present invention;

[0051] Figure 8 A schematic diagram of an adaptive projection mapping layer provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0053] Unless otherwise specifically stated, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0054] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0055] Additionally, descriptions of well-known structures, functions, and configurations may be omitted for clarity and conciseness. Those skilled in the art will recognize that various changes and modifications can be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0056] Technologies, methods and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods and equipment should be considered part of the authorization specification.

[0057] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0058] Example 1

[0059] like Figure 1 As shown, this embodiment provides a multimodal large model based on dynamic decision hybrid expert mapping, including:

[0060] The visual coding group includes several visual coding structures with different functions, which are used to perform different visual coding processing on the same original reasoning image to obtain several corresponding visual coding features;

[0061] The visual fusion and compression module is used to fuse and compress several visual coding features and output multi-scale image features;

[0062] Dynamic decision-making hybrid expert mapping module, including:

[0063] The hybrid expert module is used to select several expert modules from multiple expert modules according to the dynamic routing mechanism to process the dimensional features of different dimensions in the multi-scale image features respectively, and connect the features processed by several expert modules to obtain highly discriminative features;

[0064] The shared expert module is used to process the dimensional features of different dimensions in the multi-scale image features separately and output the comprehensive features that integrate multimodal information;

[0065] The goal is to fuse multimodal information and uniformly process the different dimensions of multi-scale image features. This design achieves cross-dimensional and cross-modal feature interaction and information integration through parameter sharing, extracting comprehensive features without significantly increasing model complexity. It is suitable for high-level tasks such as classification, detection, and segmentation, and is particularly well-suited for processing high-dimensional sparse data or multimodal correlation tasks.

[0066] Decision expert module, used to estimate the importance of each dimension of multi-scale image features;

[0067] The decision connection module is used to obtain the connection weights corresponding to the high-discriminative features and the comprehensive features based on the importance of the features of each dimension, and to connect the high-discriminative features and the comprehensive features according to their respective connection weights to obtain the decision features;

[0068] Adaptive projection mapping layer, used to map decision features to language semantic space to obtain input image features;

[0069] The large language model is used to perform event reasoning and judgment based on the input image features and the text features corresponding to the original inference image, and ultimately generate language response output.

[0070] Specifically, the raw inference images referred to in this application refer to images of various urban events captured by cameras or urban surveillance equipment mounted on mobile vehicles (such as city buses). For example, these vehicles can capture images of peddlers on city roads. In this context, peddlers are vehicles that stop on various city roads to sell goods, such as fruits, vegetables, and snacks. They can use four-wheeled trucks, three-wheeled trucks, or other cargo-carrying vehicles like three-wheeled bicycles. Their characteristic characteristic is that these vehicles, carrying goods for sale, remain in a certain location for a long period of time. In the context of peddlers, accurately detecting whether peddlers are occupying the road at their location through images would help city managers manage peddlers and prevent them from occupying the road. However, due to the complexity of peddlers, misidentification is prone to occur. For example, a vehicle carrying goods but not selling goods while in motion, simply transporting goods, could be mistakenly identified as a peddler. Alternatively, vehicles such as fruit carts temporarily parked on the road could be mistakenly identified as peddlers. In view of this situation, the prior art uses target detection to identify whether the person is a traveling peddler, but it can only provide the target category, and complex scenes cannot be distinguished or even provide wrong judgments. For this reason, it is very necessary to adopt a text description and reasoning ability that takes into account scene factors or a large model with attribute text description. Therefore, this application uses a multimodal large model to solve these problems. In addition, in order to improve the recognition accuracy, this application also proposes to improve the existing Yolov11 network structure and add a scene-specific feature extraction module, which can optimize the problem of inaccurate processing of mobile vendor scene pictures obtained in specific scenes, such as in bad weather, foggy days or strong light, and improve the model's processing accuracy for pictures taken in these specific scenes.

[0071] The following is an explanation of each of the above modules:

[0072] 1. Visual encoding group

[0073] The visual coding group includes: an original visual coding structure, which is used to receive the original reasoning picture for original visual coding and output the original visual coding features; an enhanced visual coding structure, which is used to receive the original reasoning picture for enhanced image coding and output enhanced visual coding features; a proprietary visual coding structure, which is used to receive the original reasoning picture for proprietary visual image coding and convert it, and output proprietary visual coding features that are consistent with the dimensions of the original visual coding features and enhanced visual coding feature structures.

[0074] 1.1. Enhanced visual encoding module or original visual encoding structure

[0075] The enhanced visual coding module or the original visual coding structure basically uses a VIT structure model, which in turn uses a transformer-based structure. For example, the enhanced visual coding structure uses the SAM2 model. The existing transformer structure proposes an encoder-decoder structure, while the underlying structure uses a transformer and FFN structure. As can be seen from the figure, the transformer structure consists of Q, K, and V, where Q, K, and V are implemented using conventional linear convolution. The transformer structure is finally implemented using the following formula 1, which is calculated as follows:

[0076]

[0077] where d k Represents the feature expression dimension.

[0078] The FFN structure consists of two linear convolutions and a ReLU activation function, and its calculation formula is as follows:

[0079] FFN(x)=max(0,xW1+b1)w2+b2 (2)

[0080] 1.2 Proprietary Visual Coding Structure

[0081] like Figure 6 As shown in the figure, the proprietary visual encoding structure is an improvement on the Yolov11 network structure. It is used to solve conditions such as lighting and defogging. It is a method for obtaining proprietary visual features for urban management. This is a feature information expression that the multimodal and enhancement modules do not have. Specifically, it includes:

[0082] 1.2.1 Decision Module

[0083] The decision module is used to make a binary classification decision on the received original reasoning image to determine whether it is necessary to further extract scene-specific features. If the judgment result is yes, the original reasoning image is input into the scene-specific feature extraction module and the Yolov11 network structure respectively. If the judgment result is no, the original reasoning image is directly input into the Yolov11 network structure. The decision module uses the resnet50 classification model as the decision model.

[0084] 1.2.2 Scenario-specific Feature Extraction Module

[0085] The scene-specific feature extraction module is used to extract the specific features of the original inference image and input the first feature into the backbone network of the Yolov11 network structure; Figure 6As shown, the four-layer structure of the scene-specific feature extraction module is: two identical layers of tandem Conv modules, a layer of C3K2 modules, and a final Conv module. The two identical layers of tandem Conv modules and the C3K2 module are used to extract specific visual features from the original inference image. The final Conv module is used to match the specific visual features with the feature dimensions of the Yolov11 network structure and output the first feature. The first three layers of the scene-specific feature extraction module are identical to the first three layers of the Yolov11 network backbone network. The scene-specific feature extraction module is embedded between the first three layers of the Yolov11 backbone network and the fourth layer.

[0086] In order to enhance the feature adaptability structure, in the last layer Conv module structure, Conv k=1, s=1 is the adjusted feature, the size of the convolution kernel is 1, and the step size of the convolution operation is 1. The last layer Conv module adjusts the dimension of the proprietary visual feature to be consistent with the second feature dimension output by the C3K2 module of the backbone network, outputs the first feature (if the decision module determines that the scene-specific feature extraction module needs to be used, the feature is output), and then uses the cat structure to directly add the first and second features for fusion.

[0087] 1.2.3 Yolov11 Network Structure

[0088] The Yolov11 network structure in this embodiment is the same as the existing structure, except that a scene-specific feature extraction module is embedded between the first three layers and the fourth layer to accurately identify images in specific scenes, such as Figure 6 As shown in the figure, the Yolov11 network structure includes a backbone network (Backbone), a neck network (Neck) and a head network (Head), wherein the backbone network is embedded with a scene-specific feature extraction module, and decides whether to fuse the first feature output by the scene-specific feature extraction module based on the judgment result of the decision module.

[0089] 2. Visual Fusion and Compression Module

[0090] The visual fusion and compression module includes: a dynamic threshold module, which includes several dynamic threshold gating networks corresponding to each visual coding structure, each dynamic threshold gating network is used to adaptively assign a weight value to each visual coding feature;

[0091] A feature fusion module is used to fuse several visual coding features according to the weight value of each visual coding feature and output a fused feature;

[0092] The feature compression module is used to compress the fused features and output multi-scale image features.

[0093] like Figure 3 As shown, for the visual coding features output by the three visual coding structures of the visual coding group, specifically, the original visual coding features are obtained from the original visual coding structure, and their format is (BLC), the enhanced visual coding features are obtained from the enhanced visual coding structure, and their format is (BL1C1), and the proprietary visual coding features are obtained from the proprietary visual coding structure, and their format is (BC3H3W3). In order to make the proprietary visual coding features consistent with the formats of the original visual coding features and the enhanced visual coding features, an adaptive visual feature conversion structure is provided between the hybrid visual feature coding group and the visual group fusion and compression module, which converts the formats of the proprietary visual coding features, the original visual coding features, and the enhanced visual coding features into the same format. The adaptive visual feature conversion structure includes three MLP feature conversion modules, and the three MLP feature conversion modules process the proprietary visual coding features, the original visual coding features, and the enhanced visual coding features, respectively. The proprietary visual coding features are converted by first transforming (B C3 H3 W3) into (B C3 H3*W3) through format transformation, and then performing position transformation (BH3*W3 C3) to finally obtain (B L3 C3), where L3 = H3*W3.

[0094] As can be seen, the output format of the original visual coding structure, the enhanced visual coding structure, and the proprietary visual coding structure are all [BLC], where B represents the input image batch, L represents the token, and C represents the dimensional vector representation of each token. BLC structure data is required to input the dynamic decision hybrid expert mapping module. To adapt to different scene requirements and dynamically match scenes, the visual fusion and compression module in this application dynamically assigns different weights to each visual coding feature to more effectively integrate the fused feature representation.

[0095] Three dynamic threshold gating networks are set up corresponding to the three visual coding structures. The weight value of each visual coding feature is determined, and the residual of each visual coding feature and its corresponding weight value is calculated and input into the feature fusion module. Then, the feature fusion module connects the feature values ​​output by the three dynamic threshold gating networks and inputs them into the feature compression module for compression and output. Specifically:

[0096] 2.1 Dynamic Threshold Module

[0097] like Figure 4As shown, the dynamic threshold module includes three dynamic threshold gating networks: a first dynamic threshold gating network for determining the weight value of the original visual coding feature, a second dynamic threshold gating network for determining the weight value of the enhanced visual coding feature, and a third dynamic threshold gating network for determining the weight value of the proprietary visual coding feature. Each dynamic threshold gating network includes a multi-head attention structure, a compression structure, and a weight distribution structure connected in sequence. Specifically:

[0098] 2.1.1 Multi-head Attention Block

[0099] The multi-head attention structure is used to extract multi-scale features from the input visual encoding features; Figure 5 As shown, the multi-head attention structure includes a first normalization layer LayerNorm, a second multi-head attention layer Mult-HeadAttention, a third normalization layer LayerNorm and a fourth feedforward neural network layer FFN connected in sequence, wherein the output of the first normalization layer and the output of the second multi-head attention layer are input into the third normalization layer after residual connection, and the output of the third normalization layer and the output of the fourth feedforward neural network layer are used as the output of the multi-head attention structure after residual connection.

[0100] Since the format of each visual coding feature is the same, all are (BLC), B represents the batch of the original input image, L represents the token, and C represents the dimensional expression of each token. Before inputting the multi-head attention structure, each dynamic threshold gating network needs to input the visual coding features and perform corresponding position encoding on the input features to obtain a feature format that conforms to the multi-head attention structure input. For example, after the original visual coding feature Fo(B, L, C) is input into the multi-head attention structure of the first dynamic threshold gating network, the multi-scale feature (B, L, C) is obtained. After the enhanced visual coding feature Fe(B, L, C) is input into the multi-head attention structure of the first dynamic threshold gating network, the multi-scale feature (B, L, C) is obtained. After the proprietary visual coding feature Fp(B, L, C) is input into the multi-head attention structure of the first dynamic threshold gating network, the multi-scale feature (B, L, C) is obtained.

[0101] 2.1.2 Compression Structure

[0102] The compression structure is used to compress multi-scale features into image coding feature outputs of uniform dimension. The image coding feature output dimensions of the compression structure of each dynamic threshold gating network are consistent.

[0103] Specifically, the compression structure includes a first and second compression layer connected in sequence. Both the first and second compression layers include a linear layer (Linear), a normalization layer (LN), and an activation layer (RELU). The first compression layer compresses the dimensionality of the multi-scale feature layer to 1 / 2 of its original size, and the second compression layer further compresses the dimensionality of the multi-scale feature layer to 1.

[0104] It can be seen that after the multi-scale feature (B, L, C) passes through the first layer of compression structure, the feature dimension becomes: (B, L, C / / 2), and after passing through the second layer of compression structure, the feature dimension becomes: (B, L, 1). The compression modules of the three dynamic threshold gating networks output Fo1(B, L, 1), Fe1(B, L, 1), and Fp1(B, L, 1) respectively.

[0105] 2.1.3 Weight Distribution Structure (Softmax Function)

[0106] The weight distribution structure is used to calculate the proportion of the current input image coding feature in the sum of the image coding features output by the compression structure of all dynamic threshold gating networks, and obtain the weight value corresponding to the current input visual coding feature.

[0107] For example, the weight value output by the first dynamic threshold gating network is μ, the weight value output by the second dynamic threshold gating network is ρ, and the weight value output by the third dynamic threshold gating network is τ, where μ+ρ+τ=1. The weight value calculation process corresponding to each visual coding feature is:

[0108] The weight value of the original visual encoding feature:

[0109] Enhanced visual encoding features:

[0110] Proprietary visual encoding features:

[0111] 2.2 Feature Fusion Module

[0112] The feature fusion module is used to fuse several visual coding features according to the weight value of each visual coding feature and output the fusion feature F = Concat(μFo, ρFe, τFp);

[0113] 2.3 Feature Compression Module

[0114] The feature compression module consists of a compressed linear layer (nn.linear), a normalization function (LN), and an activation function (RELU). The compressed linear layer compresses the fused features and extracts effective features, resulting in valid features. The normalization function and activation layer perform normalization and linear transformation on the effective features, respectively, to obtain multi-scale image features. Feature compression using the nn.liear method reduces high-dimensional features to low dimensions through linear mapping, removing redundant information and focusing on key features and achieving a feature format that satisfies the dynamic decision-making hybrid expert mapping module. For example, the 768-dimensional features output by ViT are compressed to 50 dimensions. Normalization and activation functions are then applied. Normalization normalizes the distribution of input data at each layer, accelerating training, stabilizing the neural network, and improving model performance. The activation function introduces nonlinear transformations, enabling the neural network to learn and represent complex patterns, ultimately outputting multi-scale image features.

[0115] 3. Dynamic Decision-Making Hybrid Expert Mapping Module

[0116] 3.1 Hybrid Expert Module

[0117] The hybrid expert module includes: a routing gating network, which is used to determine the expert modules involved in dimensional feature processing and match the dimensional features of each dimension of the multi-scale image feature with multiple expert modules; multiple expert modules, each expert module processes the dimensional features assigned from the routing gating network as an independent sub-network;

[0118] The connection module is used to aggregate the processed dimensional features output by each expert module into a unified feature expression to obtain highly discriminative features.

[0119] like Figure 2 As shown in the figure, the hybrid expert structure includes n expert modules (expert 1 to expert N), each of which is an independent sub-network or functional module responsible for processing a specific data mode or task. Each expert module learns the features of a specific modality (such as text, image, speech) or cross-modality (such as text-image joint reasoning) through training, and collaborates to complete complex tasks under the scheduling of the gated network. The hybrid expert module determines which experts the visual features are assigned to through routing. A learnable routing network (G) determines which part of the input is sent to which experts (E), as shown in the following formula:

[0120]

[0121] Where i represents the i-th expert module, n represents the number of n expert modules; E i(x) represents the dimensional feature token value selected by the i-th expert to process the routing gating network assignment, and G(x) represents the normalized weight value output by the routing network after normalizing each dimensional feature token value x through n expert normalization operations respectively.

[0122] To promote the stability and generalization of the model and prevent it from falling into local optimal results, among them, the routing gating network adopts the gating network method with added noise, and the gating function with noise is a simple network with a softmax function. The routing gating network enables idle experts to have a certain probability of participating in feature processing. Specifically, first add noise and linear convolution to each dimensional feature token value x, and obtain the first weight H(x) of the n expert modules assigned to each dimensional feature token value, and the representation is as follows:

[0123] H(x) i =(W g x) i +StandardNormal()·Softplus((W noise x) i )

[0124] Among them, StandardNormal() represents the standard normal distribution, and both W g and W noise represent linear convolution weights.

[0125] H(x) obtains the expert weights through adding noise and linear convolution. Select k expert modules from the n expert modules. For the selected k expert modules, that is, n expert weights are assigned to each token. For the expert selection of k (k < n), if the values of the experts satisfying the first k values are retained, otherwise they are set to -∞, and then become 0 after passing through the softmax function. The specific formula expression is as follows:

[0126]

[0127] Then, apply the softmax function to determine which experts each token is assigned to. Through the above formula, the first weight H(x) of each token expert score and the selected k experts are given, and then the n experts corresponding to each dimensional feature token value x are normalized, so that the output values of the n - k unselected experts become 0, indicating that this token does not require these expert decisions, and finally obtain the normalized weight value G(x):

[0128] G(x) i =Softmax(TopK(H(x),k)).

[0129] 3.2. Shared Expert Modules

[0130] In traditional routing strategies, tokens assigned to different experts may contain common knowledge or information. Different experts may incorporate this common knowledge into their respective parameters, leading to redundant expert parameters. Having a dedicated shared expert to capture and integrate common contextual knowledge alleviates parameter redundancy among other routing experts. This reduction in redundant parameters allows more specialized experts to build more parameter-efficient models. A one-layer MLP architecture is used.

[0131] 3.3 Decision-making Expert Structure

[0132] For different scenario tasks, whether shared experts play a greater role independently, or mixed experts work better independently, or even directly combined, these uncertainties may weaken the capabilities of mixed and shared experts, which is not conducive to model effectiveness. To this end, this application designs a decision expert module that allows users to customize how to allocate these experts. This module is not a simple weight content, but rather an importance valuation for each feature, which contributes to the decision allocation of shared expert features and mixed expert features.

[0133] The decision expert module includes: a layer of decision experts using an MLP structure, which is used to estimate the importance of each dimension feature of the multi-scale image features and obtain a predicted estimate in the format of (BL 2C); where B represents the original inference image batch, L represents the dimension feature token, and C represents the dimensional representation of each token;

[0134] A shape change module is used to transform the predicted estimate to obtain a transformed estimate in the format of (BLC 2);

[0135] The softmax activation function is used to process the conversion estimate to obtain the importance estimate of each visual feature.

[0136] For example, assuming that the format of the multi-scale image feature F6 is (BLC), F6 obtains the comprehensive feature F61 after passing through the shared expert module, and the format is still (BLC). Input F6 obtains the feature F62 through the decision expert module, and the format is (BLC, 2). This is the importance estimation of each dimension feature of the multi-scale image feature. Input F6 to the hybrid expert module to obtain the high discriminant feature F63, and the format is still (BLC). After passing through the decision connection module, the F63 and F61 features are feature-connected by the importance estimation of F63 and F61 obtained by the decision F62, and the decision feature = F 61 ·F 62 […,0]+F 63 ·F 62 […,1]. F 62[…,0] and F 62 […,1] represents the importance estimation corresponding to each dimension feature.

[0137] 3.4 Adaptive Projection Mapping Layer

[0138] The adaptive projection mapping layer adopts a multi-layer perception MLPs structure, such as Figure 8 As shown in the figure, the multi-layer perception MLPs structure is obtained by multiple cycles of a multi-layer perceptron MLP module, and an MLP structure with residual connection is added, which specifically includes two fully connected layers and an activation function, as shown in the formula below.

[0139] M(x)=x+[W2(σ(W1*x+b1))+b2]

[0140] Among them, σ(·) is the activation function, and this application chooses RELU as the activation function; W1 and W2 represent nn.Linear linear weights, respectively, and b1 and b2 are the corresponding biases. A multi-layer perception MLPs structure is adopted, so that the input decision features are passed through multiple cycles Nx and then connected with the output of the second fully connected layer through a residual connection to output the input image features that meet the input of the large language model. Since the spatial state and feature dimension of multi-scale image features are higher than those of text features, in order to enable the splicing of multi-scale image features and text features in the large language model, it is necessary to map the multi-scale image features to the language semantic space so that they are consistent with the state of the text features.

[0141] 4. Large Language Model

[0142] This embodiment is an improvement based on the LLaVA multimodal large model. The LLaVA multimodal large model adopts the CLIP visual feature network as the visual encoder. The multi-scale image feature XvImage after fusion and compression by the visual group fusion and compression module is converted into a language embedding token Hv in the word embedding space through the dynamic decision hybrid expert mapping module. The text input instruction Xq language Instruction is converted into a text feature Hq through the text instruction encoding structure. Hv and Hq are passed through the original large language model Language Model of the LLaVA multimodal large model to generate a language response LanguageResponse Xa output.

[0143] In order to obtain the modules mentioned above, it is necessary to collect a large number of pictures of urban mobile vendors to train various scenes, such as Figure 7 As shown, for example, the training process for the mobile vendor scenario includes the following steps:

[0144] Step 1: Proprietary visual feature extraction training

[0145] S11. Construct a training dataset. Use the vehicle-mounted device to collect several pictures of mobile vendors related to urban management and store them in the training dataset, which is recorded as data1.

[0146] S12. Filter out mobile vendor scene images corresponding to proprietary scenarios (proprietary scenarios include bad weather, foggy days, or strong light scenes) from data1 and label these mobile vendor scene images with "yes" and save them in a proprietary image dataset, recorded as data2. Label the mobile vendor scene images in the difference set of data1 and data2 with "no" and then combine the labeled data2 and the difference set into an image decision label dataset, recorded as data3.

[0147] S13, the training is divided into auxiliary decision training, first-stage training and second-stage training. The auxiliary decision training uses data3 to train the decision module. The decision module selects the resnet50 classification model to obtain the optimal binary target detection decision model and the optimal weight weight3 of the model. The optimal binary target detection decision model is recorded as model3.

[0148] S14, the first stage of training is to use data1 to train the Yolov11 network structure to obtain the best target detection model and its weight weight1, and the best target detection model is recorded as model1;

[0149] S15. Use the same module as Yolov11 to build a proprietary feature extraction module. This module has two Convs and one C3K2 for special processing of proprietary scenes. The last Conv is used to extract proprietary scene features that adapt to the original model feature space of Yolov11. Finally, use the cat structure to directly add them together for fusion. The second stage of training is to use data2 to train the scene-specific feature extraction module. At the same time, use weight1 to assign values ​​to the Yolov11 network structure during training. When training the scene-specific feature extraction module, freeze all structures of the Yolov11 network structure and only train the scene-specific feature extraction module to obtain the optimal extraction model, recorded as model2, and its corresponding optimal weight weight2;

[0150] S16. According to weight1, weight2, and weight3, the weights of the Yolov11 network structure, the scene-specific feature extraction module, and the decision module are adjusted respectively to obtain a trained proprietary visual encoding structure.

[0151] Step 2: Proprietary Visual Feature Extraction, Embedding, and Inference

[0152] The optimal models corresponding to the Yolov11 network structure, scene-specific feature extraction module, and decision module obtained above are embedded in the system. The specific process is as follows:

[0153] S21. Build a proprietary visual encoding structure. The integrated decision module + scene-specific feature extraction module is embedded in the Yolov11 network structure, such as Figure 6 As shown in the figure, the weights trained above are assigned to each structure to obtain a complete vision-specific feature extraction network. The original trained head module has been removed here, and the features are directly output in the format of (B C3 H3 W3).

[0154] S22, binary classification decision model reasoning. Input the reasoning image I and perform a binary classification decision to determine whether further extraction of unique visual features is required. If yes, the decision is yes; otherwise, the decision is no.

[0155] S23: Dedicated visual feature reasoning. If the decision in step S22 is yes, the inference image I is input into the scene-specific feature extraction module for reasoning, and the final output is the first feature F1; if the decision in step S22 is no, no reasoning is performed.

[0156] S24: Inference of proprietary visual feature embedding. Image I is processed through the first three layers of Yolov11 (Conv-Conv and C3K2) to obtain the second feature F2. If the decision in step S22 is yes, features F1 and F2 are directly added together to form the fused feature F3. If the decision in step S22 is no, feature F3 is equal to feature F2.

[0157] S25, model inference. After obtaining F3 in step S24, continue to complete the processing of the remaining structure of Yolov11, and finally obtain the proprietary visual encoding feature output, whose format is (B C3 H3 W3)

[0158] Step 3: Training of multimodal large models

[0159] S31. Construct a dataset data4. The data in this dataset corresponds to the multimodal sample data annotation of urban management. It is a text annotation of an image and multiple rounds of dialogue.

[0160] S32. Train the LLaVA multimodal model. Input data4 into the LLaVA multimodal model for model training to obtain the optimal model. While the LLaVA multimodal model is training, the structure in the visual encoding group needs to be frozen, and no weight update is performed.

[0161] S33. Model deployment. The proprietary visual encoding structure obtained in step 1 and the optimal model obtained in step S32 are deployed in a cloud server and applied to smart city-related tasks.

[0162] It is understandable that this application constructs a visual feature coding group based on a multimodal large model. In addition to retaining the original visual feature extraction, it adds additional enhanced visual coding to further improve the generalization ability of its visual features. In addition, a proprietary visual feature extraction module is provided for urban management scenarios. This module is pre-trained in existing scenes to obtain proprietary features adapted to the scene. Since the existing visual fusion and compression modules are generally fused through simple concat, which is relatively rough, it is impossible to determine which visual coding feature is more effective for the scene task. Different scenes may require more original visual features or more enhanced visual features at certain times. It is also impossible to manually set the weight value of each visual coding feature to make it meet the scene requirements. In order to solve this problem, this embodiment proposes a system that adaptively adjusts the weight of each feature coding feature to dynamically match the scene, and adjusts these parameters according to the task or scene requirements.

[0163] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Based on the technical essence of the present invention and within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement of the above embodiment shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A large multimodal model based on dynamic decision-making hybrid expert mapping, characterized by: include: The visual coding group includes several visual coding structures with different functions, which are used to perform different visual coding processing on the same original reasoning image to obtain several corresponding visual coding features; The visual fusion and compression module is used to fuse and compress several visual coding features and output multi-scale image features; Dynamic decision-making hybrid expert mapping module, including: The hybrid expert module is used to select several expert modules from multiple expert modules according to the dynamic routing mechanism to process the dimensional features of different dimensions in the multi-scale image features respectively, and connect the features processed by several expert modules to obtain highly discriminative features; The shared expert module is used to process the dimensional features of different dimensions in the multi-scale image features separately and output the comprehensive features that integrate multimodal information; Decision expert module, used to estimate the importance of each dimension of multi-scale image features; The decision connection module is used to obtain the connection weights corresponding to the high-discriminative features and the comprehensive features based on the importance of the features of each dimension, and to connect the high-discriminative features and the comprehensive features according to their respective connection weights to obtain the decision features; Adaptive projection mapping layer, used to map decision features to language semantic space to obtain input image features; The large language model is used to perform event reasoning and judgment based on the input image features and the text features corresponding to the original inference image, and ultimately generate language response output.

2. The multimodal large model based on dynamic decision hybrid expert mapping according to claim 1 is characterized in that: The decision expert module includes: a layer of decision experts using an MLP structure, which is used to estimate the importance of each dimension feature of the multi-scale image feature and obtain a predicted estimate in the format of (BL 2C); where B represents the original inference image batch, L represents the dimension feature token, and C represents the dimensional representation of each token; A shape change module is used to transform the predicted estimate to obtain a transformed estimate in the format of (BLC 2); The activation function is used to process the conversion estimate to obtain the importance estimate of each visual feature.

3. The multimodal large model based on dynamic decision hybrid expert mapping according to claim 2 is characterized in that: The hybrid expert module includes: a routing gating network for determining the expert modules involved in dimensional feature processing and matching the dimensional features of each dimension of the multi-scale image features with multiple expert modules; Multiple expert modules, each expert module processes the dimensional features assigned from the routing gating network as an independent sub-network; The connection module is used to aggregate the processed dimensional features output by each expert module into a unified feature expression to obtain highly discriminative features.

4. The multimodal large model based on dynamic decision hybrid expert mapping according to claim 3 is characterized in that: The high-discriminative feature y output by the connection module is expressed as follows: Where i represents the i-th expert module, n represents the number of n expert modules; E i (x) represents the dimension feature token value assigned by the routing gating network selected by the i-th expert, and G(x) represents the normalized weight value output after each dimension feature token value x is normalized by n experts.

5. The multimodal large model based on dynamic decision hybrid expert mapping according to claim 3 is characterized in that: The routing gating network adopts the gating network method of adding noise. First, noise and linear convolution are added to each dimension feature token value x, and the first weight H(x) of the n expert modules assigned to each dimension feature token value is obtained. It is expressed as follows: H(x)=W g x+StandardNormal()·Softplus(W noise x) Among them, StandardNormal() represents the standard normal distribution, W g With W noise Both represent linear convolution weights. Select k expert modules from n expert modules. For the selected k expert modules, the output value of the i-th expert module is retained. Use the Softmax function and the first weight H(x) to normalize the n experts corresponding to each dimension feature token value x, so that the output values ​​of the unselected nk experts become 0, and obtain the normalized weight value G(x): G(x)=Softmax(TopK(H(x),k)).

6. The multimodal large model based on dynamic decision hybrid expert mapping according to claim 1 is characterized in that: The shared expert module adopts a 1-layer MLP structure.

7. The multimodal large model based on dynamic decision hybrid expert mapping according to claim 1 is characterized in that: The adaptive projection mapping layer adopts a multi-layer perceptron (MLP) structure. The multi-layer perceptron (MLP) structure includes a multi-layer perceptron (MLP) module with multiple cycles and residual connections. The multi-layer perceptron (MLP) module includes a first fully connected layer, an activation function, and a second fully connected layer connected in sequence. The input decision features are recycled multiple times and then residually connected with the output of the second fully connected layer to output the input image features.

8. The multimodal large model based on dynamic decision hybrid expert mapping according to claim 1 is characterized in that: The visual coding group includes the original visual coding structure, the enhanced visual coding structure and the proprietary visual coding structure. The proprietary visual coding structure includes: Yolov11 network structure, scene-specific feature extraction module and decision module, among which, The decision module is used to perform a binary classification decision on the received raw reasoning image to determine whether further scene-specific features need to be extracted. If the judgment result is yes, the raw reasoning image is input into the scene-specific feature extraction module and the Yolov11 network structure respectively. If the judgment result is no, the raw reasoning image is directly input into the Yolov11 network structure; The scene-specific feature extraction module is used to extract the specific features of the original inference image and input the first feature into the backbone network of the Yolov11 network structure; The Yolov11 network structure includes a backbone network, a neck network and a head network. The backbone network is embedded with a scene-specific feature extraction module, and decides whether to fuse it with the first feature output by the scene-specific feature extraction module based on the judgment result of the decision module.

9. The multimodal large model based on dynamic decision hybrid expert mapping according to claim 8, characterized in that: The structure of the scene-specific feature extraction module is the same as the first four layers of the backbone network of the Yolov11 network structure. The scene-specific feature extraction module is embedded between the first three layers and the fourth layer of the backbone network of the Yolov11 network structure.

10. The multimodal large model based on dynamic decision hybrid expert mapping according to claim 1, characterized in that: The visual fusion and compression module includes: a dynamic threshold module, which includes several dynamic threshold gating networks corresponding to each visual coding structure, each dynamic threshold gating network is used to adaptively assign a weight value to each visual coding feature; A feature fusion module is used to fuse several visual coding features according to the weight value of each visual coding feature and output a fused feature; The feature compression module is used to compress the fused features and output multi-scale image features.

Images