A multi-expert voting crowd counting method based on scene understanding

Abstract

The present invention belongs to the field of image analysis technology, and specifically relates to a multi-expert voting crowd counting method based on scene understanding, the steps of which include using a backbone network including an image encoder, a text encoder and a feature enhancer to extract visual features from an input image and extract text features from a text prompt; projecting the text features into the same embedding space as the visual features for alignment; constructing a local subject expert for local crowd target detection, a global context expert for understanding the overall layout of a scene, a spatial relationship expert for modeling the spatial relationship between crowds, and an attribute feature expert for processing the appearance and attribute information of targets in the scene; adopting a knowledge distillation strategy to train each expert in fine granularity; obtaining the output of each expert, and outputting the final counting result through a decoder. The present invention can optimize the fine-grained crowd counting performance in complex scenes by using the voting mechanism and knowledge distillation strategy of a multi-expert system.

Description

Technical Field

[0001] The present invention belongs to the technical field of image analysis, and in particular relates to a multi-expert voting crowd counting method based on scene understanding. Background Art

[0002] With the rapid development of computer vision technology, crowd counting plays an increasingly important role in public safety monitoring, intelligent transportation, and business analysis. Existing crowd counting methods mainly focus on counting the entire crowd. However, when fine-grained attribute feature counting is required (such as distinguishing different positions, postures, appearance attributes, etc.), the performance of these methods is usually limited. Fine-grained counting requires not only the model to recognize the category of the target, but also to distinguish individual attributes in complex scenes. Current methods still have certain difficulties in these aspects.

[0003] In order to improve the crowd counting effect of fine-grained attribute features in complex scenarios, technologies based on multimodal information fusion and multi-expert systems have gradually attracted attention in recent years. Multimodal information fusion (such as the combination of visual and textual information) helps the model extract richer fine-grained features, while multi-expert systems can treat different attribute features as independent processing units, making the model more sophisticated and accurate when processing different types of information. However, there is currently a lack of a counting method that can combine the two well. Summary of the invention

[0004] In view of the above deficiencies in the prior art, the purpose of the present invention is to provide a multi-expert voting crowd counting method based on scene understanding, which can utilize the voting mechanism and knowledge distillation strategy of the multi-expert system to optimize the fine-grained crowd counting performance in complex scenarios.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is a multi-expert voting crowd counting method based on scene understanding, comprising the following steps:

[0006] S1. Extract visual features, including shallow visual features and deep visual features, from the input image using a backbone network including an image encoder, a text encoder, and a feature enhancer, optimize the shallow visual features and the deep visual features, and extract text features, including local text features and global text features, from text prompts;

[0007] S2, project the text features into the same embedding space as the visual features for alignment;

[0008] S3. Construct a local subject expert for local crowd target detection, whose input is local text features (aligned, the same below) and shallow visual features (optimized, the same below), and the output is the counting result of the crowd in the local area;

[0009] S4. Build a global context expert for understanding the overall layout of the scene, whose input is global text features and deep visual features, and output is a panoramic crowd counting prediction;

[0010] S5. Construct a spatial relationship expert for modeling the spatial relationship between people, whose input is shallow visual features and deep visual features, and output is a multi-level association of shallow visual features and deep visual features;

[0011] S6, construct an attribute feature expert for processing the appearance and attribute information of the target in the scene, whose input is shallow visual features and global text features, and output is attribute counting results;

[0012] S7, adopt the knowledge distillation strategy to train each expert in a fine-grained manner;

[0013] S8. Obtain the output of each expert, use a multi-layer perceptron gated network, fuse the output of each expert through a weighted voting mechanism, and output the final counting result through a decoder.

[0014] As a preferred embodiment of the present invention, in S1, the process of extracting visual features is as follows:

[0015] S1.1. For the input image I, the image encoder extracts multi-scale visual features through Swin-Transformer, which is expressed as:

[0016] (1);

[0017] In the formula, , Represent shallow visual features and deep visual features respectively;

[0018] S1.2, through the feature enhancer and To perform feature enhancement, first extract the visual features in the horizontal direction h and the vertical direction v, expressed as:

[0019] (2);

[0020] In the formula, Represents the decomposition result of shallow visual features, including shallow visual features in the horizontal direction and shallow visual features in the vertical direction ; Represents the decomposition result of deep visual features, including deep visual features in the horizontal direction and vertical deep visual features ;

[0021] S1.3, through cross-attention, information interaction between horizontal and vertical directions is carried out to optimize shallow and deep visual features, which can be expressed as:

[0022] (3);

[0023] In the formula, , They represent the optimized shallow visual features and deep visual features respectively.

[0024] As a preferred embodiment of the present invention, in S1, based on a text encoder including a tokenizer and a pre-trained deep learning model BERT, the process of extracting text features is as follows:

[0025] S1.4. For the text prompt T, the text encoder converts it into a semantic vector through Tokenizer, which is expressed as:

[0026] (4);

[0027] In the formula, , Represent the local text embedding vector and the global text embedding vector respectively;

[0028] S1.5. Extraction via BERT , The corresponding text features are expressed as:

[0029] (5);

[0030] In the formula, It is a local text feature that captures the semantic information of a specific target; It is a global text feature that focuses on contextual semantic information.

[0031] As a preferred solution of the present invention, in S2, a linear transformation is introduced , projecting the text features into the same embedding space as the visual features for alignment, expressed as:

[0032] (6);

[0033] In the formula, is a nonlinear activation function; , They represent the aligned local text features and global text features respectively.

[0034] As a preferred solution of the present invention, in the above S3, in order to make the local subject expert focus on the foreground head area, the local irrelevant information is filtered out by using the regional mask. The background area is shielded, expressed as:

[0035] (7);

[0036] In the formula, It is the shallow visual feature after filtering; represents the Hadamard product; By correlation probability Decide, is the threshold value;

[0037] The local subject expert uses Q-Former to cross-modally fuse local text information with shallow visual features, which can be expressed as:

[0038] (8);

[0039] In the formula, Represents the output features after fusion of local subject experts; Represents learnable local query features.

[0040] As a preferred solution of the present invention, in S4, a pyramid pooling unit is introduced to The information integration at different scales is expressed as:

[0041] (9);

[0042] In the formula, Indicates a pooling layer with a downsampling multiple of n; Up indicates an upsampling operation; Concat indicates a concatenation operation; It is the deep visual features of different scales integrated by the pyramid pooling unit;

[0043] The global context expert uses Q-Former to cross-modally fuse global text features with deep visual features, expressed as:

[0044] (10);

[0045] In the formula, Represents the output features after global context expert fusion; represents a learnable global query feature.

[0046] As a preferred solution of the present invention, in S5, the spatial relationship expert performs multi-level correlation modeling on the shallow visual features and the deep visual features to improve the accuracy of counting; a dynamic feature reweighting mechanism and gate control are introduced to perform category-related reweighting on the input features, generate a weight matrix, and adjust the attention intensity of different feature streams, which is expressed as:

[0047] (11);

[0048] In the formula, is the generated dynamic weight; is the activation function; MLP is a multi-layer perceptron; Concat represents a concatenation operation;

[0049] The output of the spatial relationship expert is expressed as:

[0050] (12);

[0051] In the formula, Represents the features after spatial relationship experts fusion; , is a learnable parameter.

[0052] As a preferred solution of the present invention, in S6, the attribute feature experts are respectively and Attention interaction is performed to learn attribute features of fine-grained and global semantic fusion; The query vector for interaction is defined as , then The interaction is expressed as:

[0053] (13);

[0054] In the formula, Representation and Shallow visual features learned interactively; for The transpose of express The dimension size of is the activation function;

[0055] and The query vector for interaction is defined as , then The interaction is expressed as:

[0056] (14);

[0057] In the formula, Representation and Global text features learned interactively; for The transpose of express The dimension size of

[0058] The attribute feature expert combines the learned shallow visual features with the global text features based on Q-Former, focuses on the targets of the specified attributes, including color, size or posture, and makes fine distinctions on the counts of the targets, providing distribution statistics based on the target appearance, expressed as:

[0059] (15);

[0060] In the formula, represents the fusion output of attribute feature experts; Represents a learnable query output.

[0061] As a preferred solution of the present invention, in S7, the process of fusing the outputs of various experts through a weighted voting mechanism is as follows:

[0062] S7.1. Dynamically adjust the fusion weight based on the error of each expert to adapt to the performance changes of different experts, expressed as:

[0063] (16);

[0064] In the formula, represents the error corresponding to the jth expert, j=1, 2, 3, 4; is the weight corresponding to the jth expert; exp is an exponential function, which represents the logarithm with the natural constant e as the base;

[0065] S7.2. Output after fusion by multi-layer perceptron gated network It is expressed as:

[0066] (17);

[0067] In the formula, is a nonlinear activation function; represents the output corresponding to the jth expert; Represents the weight bias corresponding to the j-th expert.

[0068] As a preferred solution of the present invention, in the above S8, the knowledge distillation strategy transfers the knowledge of the teacher model to the student model to achieve a more efficient reasoning effect. The teacher model adopts multiple independent networks to guide the training of each expert in a fine-grained manner to ensure that different experts can independently extract features and focus on detailed information. The student model adopts a Q-Former network with shared weights.

[0069] During the distillation process, the teacher model guides the student model to learn the fine-grained feature recognition ability of the teacher model under a simplified structure; in the inference stage, the student model performs crowd counting prediction in complex scenes, which is expressed as:

[0070] (18);

[0071] In the formula, Represents the knowledge distillation loss, which is used to measure the prediction value of the teacher model The predicted value of the student model The squared error between .

[0072] The algorithm involved in the present invention can be executed by an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the above algorithm calculation is implemented by executing the software by the processor.

[0073] The beneficial effects of the present invention are:

[0074] In the fine-grained crowd counting task with different attributes, the present invention effectively improves the accuracy and robustness of counting through scene understanding and multimodal information fusion; the introduction of knowledge distillation enables the student model to maintain a simplified structure while ensuring performance, further improving the efficiency and adaptability of the system, thereby better meeting the crowd counting needs in complex scenarios, and realizing the use of the voting mechanism of multiple expert systems and the knowledge distillation strategy to optimize the fine-grained crowd counting performance in complex scenarios.

[0075] The present invention realizes accurate counting of fine-grained people in complex scenes through multimodal feature fusion, multi-expert model processing and voting mechanism. By introducing local subject experts, global context experts, spatial relationship experts and attribute feature experts, it is possible to analyze people with different attributes in the image from multiple angles to achieve refined counting. At the same time, the knowledge distillation strategy enables the student model to still have excellent fine-grained recognition and counting capabilities under a streamlined structure. The present invention has a wide range of application value in scenarios such as public safety monitoring, intelligent transportation, and commercial analysis, and can effectively improve counting accuracy and resource utilization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a schematic diagram of the process of the present invention;

[0077] Figure 2 It is an exemplary counting flow chart in an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The embodiments of the present invention are further described below in conjunction with the accompanying drawings:

[0079] like Figure 1 As shown, a multi-expert voting crowd counting method based on scene understanding includes the following steps:

[0080] S1. Extract visual features, including shallow visual features and deep visual features, from the input image using a backbone network including an image encoder, a text encoder, and a feature enhancer, optimize the shallow visual features and the deep visual features, and extract text features, including local text features and global text features, from text prompts (such as "pedestrians on bicycles" or "pedestrians on foot");

[0081] S2, project the text features into the same embedding space as the visual features for alignment;

[0082] S3. Construct a local subject expert for local crowd target detection, whose input is local text features and shallow visual features, and output is the counting result of the crowd in the local area;

[0083] S4. Build a global context expert for understanding the overall layout of the scene, whose input is global text features and deep visual features, and output is a panoramic crowd counting prediction;

[0084] S5. Construct a spatial relationship expert for modeling the spatial relationship between people, whose input is shallow visual features and deep visual features, and output is a multi-level association of shallow visual features and deep visual features;

[0085] S6, construct an attribute feature expert for processing the appearance and attribute information of the target in the scene, whose input is shallow visual features and global text features, and output is attribute counting results;

[0086] S7, adopt the knowledge distillation strategy to train each expert in a fine-grained manner;

[0087] S8. Obtain the output of each expert, use a multi-layer perceptron gated network, fuse the output of each expert through a weighted voting mechanism, and output the final counting result through a decoder.

[0088] Shallow visual features focus on the salient features of local areas through a smaller receptive field, which helps to identify details, such as the distribution of people in a certain area or the status of a specific activity. Deep visual features have a larger receptive field and are used to capture the overall semantics and layout information of the scene.

[0089] In S1, the process of extracting visual features is as follows:

[0090] S1.1. For the input image I, the image encoder extracts multi-scale visual features through Swin-Transformer, which is expressed as:

[0091] (1);

[0092] In the formula, , Represent shallow visual features and deep visual features respectively;

[0093] S1.2, through the feature enhancer and To perform feature enhancement, first extract the visual features in the horizontal direction h and the vertical direction v, expressed as:

[0094] (2);

[0095] In the formula, Represents the decomposition result of shallow visual features, including shallow visual features in the horizontal direction and shallow visual features in the vertical direction ; Represents the decomposition result of deep visual features, including deep visual features in the horizontal direction and vertical deep visual features ;

[0096] S1.3, through cross-attention, information interaction between horizontal and vertical directions is carried out to optimize shallow and deep visual features, which can be expressed as:

[0097] (3);

[0098] In the formula, , They represent the optimized shallow visual features and deep visual features respectively.

[0099] Swin-Transformer is a model based on the Transformer architecture, specifically designed for computer vision tasks.

[0100] In the field of deep learning and computer vision, feature enhancer is a concept used to improve feature representation capabilities. It is usually a module or a series of operations that aims to enhance the expressiveness and discrimination of input features, so that the model can more effectively learn and identify important information in the data. In this embodiment, the feature enhancer further processes and optimizes the shallow and deep visual features extracted from the image encoder and the text encoder, which can be implemented based on convolutional neural network CNN combined with known technologies such as cross attention.

[0101] In S1, based on the text encoder including the tokenizer and the pre-trained deep learning model BERT, the process of extracting text features is as follows:

[0102] S1.4. For the text prompt T, the text encoder converts it into a semantic vector through Tokenizer, which is expressed as:

[0103] (4);

[0104] In the formula, , Represent the local text embedding vector and the global text embedding vector respectively;

[0105] S1.5. Extraction via BERT , The corresponding text features are expressed as:

[0106] (5);

[0107] In the formula, It is a local text feature that captures the semantic information of a specific target; It is a global text feature that focuses on contextual semantic information.

[0108] In the field of natural language processing (NLP) and deep learning, Tokenizer and BERT are two key components in text encoders. Tokenizer is a tool that converts a text string into a series of tokens, and BERT is a pre-trained deep learning model for generating representations of text.

[0109] In S2, linear transformation is introduced , projecting the text features into the same embedding space as the visual features for alignment, expressed as:

[0110] (6);

[0111] In the formula, is a nonlinear activation function (sigmoid activation function); , They represent the aligned local text features and global text features respectively.

[0112] Introducing a linear transformation usually refers to using linear algebra methods in data processing and machine learning models to change the representation of data.

[0113] In S3, in order to make the local subject expert focus on the foreground head area and filter out the local irrelevant information, the regional mask is used The background area is shielded, expressed as:

[0114] (7);

[0115] In the formula, It is the shallow visual feature after filtering; represents the Hadamard product; By correlation probability Decide, is the threshold value;

[0116] The local subject expert uses Q-Former to cross-modally fuse local text information with shallow visual features, which can be expressed as:

[0117] (8);

[0118] In the formula, Represents the output features after fusion of local subject experts; Represents learnable local query features.

[0119] Region masking is a technique used in image processing and computer vision to mask or hide specific regions in an image. It can be obtained through learning data and used to decide which areas should be paid attention to and which should be ignored. Hadamard product is an operation that multiplies corresponding elements in an array or matrix. Q-Former (Querying Transformer) is a neural network architecture that focuses on improving information retrieval and representation learning through query mechanism.

[0120] In S4, the pyramid pooling unit is introduced to The information integration at different scales is expressed as:

[0121] (9);

[0122] In the formula, Indicates a pooling layer with a downsampling multiple of n; Up indicates an upsampling operation; Concat indicates a concatenation operation; It is the deep visual features of different scales integrated by the pyramid pooling unit;

[0123] The global context expert uses Q-Former to cross-modally fuse global text features with deep visual features, expressed as:

[0124] (10);

[0125] In the formula, Represents the output features after global context expert fusion; Represents learnable global query features. Global context features are obtained by integrating information at different scales. Cross-modal fusion with layer-by-layer alignment. The layer-by-layer alignment mechanism is used to step-by-step fuse text and visual features at different semantic levels, refining the hierarchical nature of feature fusion.

[0126] The pyramid pooling unit is a technology for capturing multi-scale features. It generates a series of feature representations of different resolutions by pooling feature maps at different scales. These feature representations can capture local and global information in the image.

[0127] In S5, spatial relationship experts perform multi-level correlation modeling on shallow and deep visual features to improve the accuracy of counting. Shallow visual features help capture local details, while deep visual features provide global layout information. Dynamic feature reweighting mechanism and gate control are introduced to reweight the input features in a category-related manner, generate a weight matrix, and adjust the attention intensity of different feature streams, which is expressed as:

[0128] (11);

[0129] In the formula, is the generated dynamic weight; is the activation function; MLP is a multi-layer perceptron; Concat represents a concatenation operation;

[0130] The output of the spatial relationship expert is expressed as:

[0131] (12);

[0132] In the formula, Represents the features after spatial relationship experts fusion; , is a learnable parameter. Spatial relationship experts can learn the relative positions and spatial distribution relationships between different targets to avoid counting errors caused by occlusion or excessive density.

[0133] In S6, attribute feature experts are respectively and Attention interaction is performed to learn attribute features of fine-grained and global semantic fusion; The query vector for interaction is defined as , then The interaction is expressed as:

[0134] (13);

[0135] In the formula, Representation and Shallow visual features learned interactively; for The transpose of express The dimension size of is the activation function;

[0136] and The query vector for interaction is defined as , then The interaction is expressed as:

[0137] (14);

[0138] In the formula, Representation and Global text features learned interactively; for The transpose of express The dimension size of

[0139] The attribute feature expert combines the learned shallow visual features with the global text features based on Q-Former, focuses on the targets of the specified attributes, including color, size or posture, and makes fine distinctions on the counts of the targets, providing distribution statistics based on the target appearance, expressed as:

[0140] (15);

[0141] In the formula, represents the fusion output of attribute feature experts; Represents a learnable query output.

[0142] Dynamic feature reweighting is a technique used to adjust feature importance in deep learning models, allowing the model to dynamically assign different weights to different features based on the specific characteristics of the input data or task requirements. Gating control is a technique used to regulate the flow of information in neural networks.

[0143] Each expert has a different weight for the impact of his or her task on the final counting result. The weighting mechanism of the gating network can dynamically adjust the weight of each expert according to the complexity of the scene and the prediction error. For example, in dense scenes, the weight of spatial relationship experts is increased, while in scenes with clear attribute distinctions, more attention is paid to the weight of attribute feature experts. Through this voting fusion mechanism, the final target count can be generated more accurately and robustly, further improving the accuracy and robustness of the overall count. Expert fusion is achieved through multi-layer perceptrons, which reduces the accumulation of errors in fusion.

[0144] In S7, the process of fusing the outputs of various experts through the weighted voting mechanism is as follows:

[0145] S7.1. Dynamically adjust the fusion weight based on the error of each expert to adapt to the performance changes of different experts, expressed as:

[0146] (16);

[0147] In the formula, represents the error corresponding to the jth expert, j=1, 2, 3, 4; is the weight corresponding to the jth expert; exp is an exponential function, which represents the logarithm with the natural constant e as the base;

[0148] S7.2. Output after fusion by multi-layer perceptron gated network It is expressed as:

[0149] (17);

[0150] In the formula, is a nonlinear activation function; represents the output corresponding to the jth expert; Represents the weight bias corresponding to the j-th expert.

[0151] The multilayer perceptron gated network is a neural network structure that combines the multilayer perceptron MLP and the gating mechanism. The gating mechanism is used to dynamically adjust the flow of information, thereby improving the model's sensitivity to different features and overall performance.

[0152] In S8, the knowledge distillation strategy transfers the knowledge of the teacher model to the student model to achieve more efficient reasoning. The teacher model uses multiple independent networks to guide the training of each expert in a fine-grained manner, ensuring that different experts can independently extract features and focus on detailed information. The student model uses a Q-Former network with shared weights, which significantly simplifies the model structure in design.

[0153] During the distillation process, the teacher model guides the student model to learn the fine-grained feature recognition ability of the teacher model under a simplified structure. In the inference stage, the student model performs crowd counting prediction in complex scenes, which not only meets the accuracy requirements but also improves the inference speed and resource utilization efficiency, which can be expressed as:

[0154] (18);

[0155] In the formula, Represents the knowledge distillation loss, which is used to measure the prediction value of the teacher model The predicted value of the student model The squared error between .

[0156] like Figure 2 As shown, for the input visual image (including people riding and hiking) and its text prompts, according to the counting method of this embodiment, the output result is finally obtained:

[0157] "Pedestrians on bicycles" count: 6;

[0158] Count of "pedestrians on foot": 37.

[0159] Figure 2 In the figure, the gated fusion network is the multilayer perceptron gated network, the linear mapping is the linear transformation, the extracted global context information (global text features) is “cyclist”, “hiker”, and the local text information (local text features) is “pedestrian”.

Claims

1. A multi-expert voting crowd counting method based on scene understanding, characterized by The following steps are involved: S1. Extract visual features, including shallow visual features and deep visual features, from the input image using a backbone network including an image encoder, a text encoder, and a feature enhancer, optimize the shallow visual features and the deep visual features, and extract text features, including local text features and global text features, from text prompts; S2, project the text features into the same embedding space as the visual features for alignment; S3. Construct a local subject expert for local crowd target detection, whose input is local text features and shallow visual features, and output is the counting result of the crowd in the local area; S4. Build a global context expert for understanding the overall layout of the scene, whose input is global text features and deep visual features, and output is a panoramic crowd counting prediction; S5. Construct a spatial relationship expert for modeling the spatial relationship between people, whose input is shallow visual features and deep visual features, and output is a multi-level association of shallow visual features and deep visual features; S6, construct an attribute feature expert for processing the appearance and attribute information of the target in the scene, whose input is shallow visual features and global text features, and output is attribute counting results; S7, adopt the knowledge distillation strategy to train each expert in a fine-grained manner; S8. Obtain the output of each expert, use a multi-layer perceptron gated network, fuse the output of each expert through a weighted voting mechanism, and output the final counting result through a decoder.

2. According to the multi-expert voting crowd counting method based on scene understanding in claim 1, it is characterized in that: In S1, the process of extracting visual features is as follows: S1.

1. For the input image I, the image encoder extracts multi-scale visual features through Swin-Transformer, which is expressed as: （1）； In the formula, , Represent shallow visual features and deep visual features respectively; S1.2, through the feature enhancer and To perform feature enhancement, first extract the visual features in the horizontal direction h and the vertical direction v, expressed as: （2）； In the formula, Represents the decomposition result of shallow visual features, including shallow visual features in the horizontal direction and shallow visual features in the vertical direction ; Represents the decomposition result of deep visual features, including deep visual features in the horizontal direction and vertical deep visual features ; S1.3, through cross-attention, information interaction between horizontal and vertical directions is carried out to optimize shallow and deep visual features, which can be expressed as: （3）； In the formula, , They represent the optimized shallow visual features and deep visual features respectively.

3. The multi-expert voting crowd counting method based on scene understanding according to claim 2 is characterized in that: In S1, based on the text encoder including the tokenizer and the pre-trained deep learning model BERT, the process of extracting text features is as follows: S1.

4. For the text prompt T, the text encoder converts it into a semantic vector through Tokenizer, which is expressed as: （4）； In the formula, , Represent the local text embedding vector and the global text embedding vector respectively; S1.

5. Extraction via BERT , The corresponding text features are expressed as: （5）； In the formula, It is a local text feature that captures the semantic information of a specific target; It is a global text feature that focuses on contextual semantic information.

4. The multi-expert voting crowd counting method based on scene understanding according to claim 3 is characterized in that: In S2, a linear transformation is introduced , projecting the text features into the same embedding space as the visual features for alignment, expressed as: （6）； In the formula, is a nonlinear activation function; , They represent the aligned local text features and global text features respectively.

5. The multi-expert voting crowd counting method based on scene understanding according to claim 4 is characterized in that: In the above S3, in order to make the local subject expert focus on the foreground head area and filter out the local irrelevant information, the regional mask is used The background area is shielded, expressed as: （7）； In the formula, It is the shallow visual feature after filtering; represents the Hadamard product; By correlation probability Decide, is the threshold value; The local subject expert uses Q-Former to cross-modally fuse local text information with shallow visual features, which can be expressed as: （8）； In the formula, Represents the output features after fusion of local subject experts; Represents learnable local query features.

6. The multi-expert voting crowd counting method based on scene understanding according to claim 4 is characterized in that: In the above S4, a pyramid pooling unit is introduced to The information integration at different scales is expressed as: （9）； In the formula, Indicates a pooling layer with a downsampling multiple of n; Up indicates an upsampling operation; Concat indicates a concatenation operation; It is the deep visual features of different scales integrated by the pyramid pooling unit; The global context expert uses Q-Former to cross-modally fuse global text features with deep visual features, expressed as: （10）； In the formula, Represents the output features after global context expert fusion; represents a learnable global query feature.

7. The multi-expert voting crowd counting method based on scene understanding according to claim 4 is characterized in that: In the above S5, the spatial relationship expert performs multi-level correlation modeling on the shallow visual features and deep visual features to improve the accuracy of counting; a dynamic feature reweighting mechanism and gate control are introduced to perform category-related reweighting on the input features, generate a weight matrix, and adjust the attention intensity of different feature streams, which is expressed as: （11）； In the formula, is the generated dynamic weight; is the activation function; MLP is a multi-layer perceptron; Concat represents a concatenation operation; The output of the spatial relationship expert is expressed as: （12）； In the formula, Represents the features after spatial relationship experts fusion; , is a learnable parameter.

8. The multi-expert voting crowd counting method based on scene understanding according to claim 4 is characterized in that: In S6, the attribute feature experts are respectively and Attention interaction is performed to learn attribute features of fine-grained and global semantic fusion; The query vector for interaction is defined as , then The interaction is expressed as: （13）； In the formula, Representation and Shallow visual features learned interactively; for The transpose of express The dimension size of is the activation function; and The query vector for interaction is defined as , then The interaction is expressed as: （14）； In the formula, Representation and Global text features learned interactively; for The transpose of express The dimension size of The attribute feature expert combines the learned shallow visual features with the global text features based on Q-Former, focuses on the targets of the specified attributes, including color, size or posture, and makes fine distinctions on the counts of the targets, providing distribution statistics based on the target appearance, expressed as: （15）； In the formula, represents the fusion output of attribute feature experts; Represents a learnable query output.

9. The multi-expert voting crowd counting method based on scene understanding according to claim 1 is characterized in that: In S7, the process of fusing the outputs of the experts through the weighted voting mechanism is as follows: S7.

1. Dynamically adjust the fusion weight based on the error of each expert to adapt to the performance changes of different experts, expressed as: （16）； In the formula, represents the error corresponding to the jth expert, j=1, 2, 3, 4; is the weight corresponding to the jth expert; exp is an exponential function, which represents the logarithm with the natural constant e as the base; S7.

2. Output after fusion by multi-layer perceptron gated network It is expressed as: （17）； In the formula, is a nonlinear activation function; represents the output corresponding to the jth expert; Represents the weight bias corresponding to the j-th expert.

10. The multi-expert voting crowd counting method based on scene understanding according to claim 1 is characterized in that: In S8, the knowledge distillation strategy transfers the knowledge of the teacher model to the student model to achieve more efficient reasoning. The teacher model uses multiple independent networks to guide the training of each expert in a fine-grained manner, ensuring that different experts can independently extract features and focus on detailed information. The student model uses a Q-Former network with shared weights. During the distillation process, the teacher model guides the student model to learn the fine-grained feature recognition ability of the teacher model under a simplified structure; in the inference stage, the student model performs crowd counting prediction in complex scenes, which is expressed as: （18）； In the formula, Represents the knowledge distillation loss, which is used to measure the prediction value of the teacher model The predicted value of the student model The squared error between .

Images