A method and apparatus for counting migrating populations based on distribution uncertainty and multimodal feature migration.

Abstract

This invention discloses a method and apparatus for crowd counting based on multimodal dynamic feature transfer under distribution uncertainty. The method includes: a multimodal dynamic feature transfer counting network using a ResNet101 residual network as the backbone, extracting multimodal features using a two-stream network structure, and generating a crowd probability density map; during the multimodal feature extraction process, intermodal feature information interaction is performed based on a channel dynamic interaction mechanism under distribution uncertainty; the probability density map output by the two-stream feature extraction network is obtained, and adaptive fusion of the multimodal output results is achieved through a decision-level adaptive fusion module; after obtaining the fused multimodal crowd probability density map, the density map is integrated to obtain the number of people. The apparatus includes a processor and a memory. This invention avoids the disadvantage of using only a single modality image for counting under poor imaging conditions and better utilizes multimodal features to improve crowd counting performance.

Description

Technical Field

[0001] This invention relates to the fields of multimodal image processing and crowd counting, and in particular to a method and apparatus for crowd counting based on multimodal dynamic feature migration with distribution uncertainty. Background Technology

[0002] Crowd counting in computer vision aims to estimate the number of people in social scenarios such as real-time pedestrian monitoring, traffic control, and public area pedestrian flow analysis using a crowd counting algorithm that performs statistical analysis on crowd density. Crowd density control is increasingly important for the normal functioning of human life and production, and efficient crowd counting methods can effectively solve this problem. In recent years, this task has attracted much attention from researchers in the field of computer vision. After the evolution of crowd counting methods based on detection and regression, most current methods calculate the number of people by generating pixel-level crowd density maps.

[0003] Regarding relevant methodologies, previous scene analysis work has largely relied on visible data. However, visible data can suffer from drawbacks such as varying lighting conditions and poor nighttime imaging. Conducting research solely under visible light perspectives may lead to difficulties in accurately determining crowd numbers due to the limited field of view and information occlusion in visible light images. Thermal infrared data captured by thermal infrared cameras effectively avoids the shortcomings of using only visible light data. Simultaneously, visible light data avoids the high-brightness interference present in thermal infrared data, allowing RGB-T (visible-infrared) images to complement each other and effectively perceive scenes both day and night. Currently, some methods utilize multimodal fusion to demonstrate the effectiveness of RGB-T (visible-infrared) in improving image analysis, but these methods do not consider dynamically utilizing the advantages of both RGB and T modes, failing to flexibly leverage the complementary strengths of the two modes to achieve better counting results.

[0004] Multimodal learning has garnered increasing attention in computer vision, and multimodal fusion has proven effective in leveraging the strengths of different modalities for feature complementarity. The most critical challenge in multimodal fusion is achieving optimal information complementarity while preserving the specificity of each modality. Most methods employ early input fusion or feature extraction followed by later fusion, failing to uncover deeper feature information from different modalities or to make dynamic fusion decisions tailored to the specific characteristics of each image set. How to better utilize information from RGB and T modalities for multimodal fusion has been a focus of research in recent years. Summary of the Invention

[0005] This invention provides a multimodal dynamic feature transfer crowd counting method and apparatus based on distribution uncertainty. The invention designs a multimodal interaction mechanism based on distribution uncertainty, enabling multimodal dynamic interaction along the channel dimension during feature extraction. This is achieved through bidirectional information transfer to improve the feature extraction performance of each modality. Furthermore, the invention designs an adaptive decision-level fusion strategy, allowing the trained model to overcome the influence of unpredictable imaging conditions and generate more reliable density maps, thereby improving crowd counting performance. Details are described below.

[0006] A first aspect is a method for counting multimodal dynamic feature migration populations based on distribution uncertainty, the method comprising:

[0007] The multimodal dynamic feature transfer counting network uses ResNet101 residual network as the backbone, extracts multimodal features with a two-stream network structure, and generates a population probability density map.

[0008] In the process of multimodal feature extraction, feature information exchange between modalities is carried out based on the channel dynamic interaction mechanism based on distribution uncertainty;

[0009] Obtain the probability density map of the output of the dual-stream feature extraction network, and realize the adaptive fusion of multimodal output results through the decision-level adaptive fusion module;

[0010] After obtaining the fused multimodal population probability density map, the population number is obtained by integrating the density map.

[0011] A second aspect is a multimodal dynamic feature migration population counting device based on distribution uncertainty, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to perform the method steps described in any of the first aspects.

[0012] Third aspect, a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the steps of the method described in any of the first aspects.

[0013] The beneficial effects of the technical solution provided by this invention are:

[0014] 1. This invention proposes a multimodal feature fusion crowd counting network that can achieve interactive transfer between different modal features, thereby improving the counting effect of each modality; this multimodal crowd counting method avoids the disadvantage of using only a single modality image for counting under poor imaging conditions, and better utilizes multimodal features to improve crowd counting performance;

[0015] 2. This invention designs a multimodal interaction mechanism based on distribution uncertainty. By utilizing a simple module design and the parameters of the network structure itself, it provides a more robust and flexible basis for interaction selection at the channel level, thereby guiding the interaction migration between different modalities. This mechanism can greatly improve the multimodal interaction counting effect.

[0016] 3. This invention proposes an adaptive decision-level fusion strategy to maximize the integration of effective results from two modalities, which helps to obtain more reliable results under unpredictable multimodal imaging conditions; and its feasibility was verified by experiments on the dual-light datasets RGBTCC and DroneRGBT. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall network structure;

[0018] Figure 2 A flowchart of a multimodal dynamic feature migration crowd counting method based on distribution uncertainty;

[0019] Figure 3 This is a schematic diagram of a multimodal dynamic feature migration population counting device based on distribution uncertainty. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0021] To address the technical problems existing in the background art, this invention proposes a multimodal feature fusion crowd counting network. This network can achieve interactive transfer between different modal features, thereby improving the counting effect of each modality and the overall network. The network includes a feature extraction module, a channel dynamic transfer module, and a decision-level adaptive fusion module. This multimodal crowd counting network avoids the disadvantage of using only a single modality image for counting under poor imaging conditions, and better utilizes multimodal features to improve crowd counting performance.

[0022] Example 1

[0023] This invention provides a method for counting multimodal dynamic feature migration populations based on distribution uncertainty. The method includes the following steps:

[0024] 101: The multimodal dynamic feature transfer counting network uses ResNet101 residual network as the backbone, extracts multimodal features with a two-stream network structure, and generates population probability density maps respectively;

[0025] Specifically, the structure of the two-stream feature extraction network is to add dilated convolutional layers as back-end networks on the ResNet101 basic network architecture, and output density maps through the final regression layer; and use the same pre-trained model (ResNet101) to initialize the two-stream counting network.

[0026] 102: In the process of multimodal feature extraction, feature information exchange between modalities is carried out based on the channel dynamic interaction mechanism based on distribution uncertainty;

[0027] In the ResNet101 network, after each layer (unit), the output features undergo a modal interaction through the channel dynamic transfer module. In the channel dynamic transfer module, the interaction object selection submodule first selects the object to be interacted with based on the distribution differences between corresponding channels in different modalities; then, the interaction direction determination submodule determines the direction for further intermodal interaction based on the distribution differences between the channels to be interacted with within a modality and the overall distribution. This multimodal collaboration improves the feature extraction performance of a single branch.

[0028] 103: Obtain the probability density map output by the dual-stream feature extraction network, and realize the adaptive fusion of multimodal output results through the decision-level adaptive fusion module;

[0029] In this process, the probability density map output by the two-stream network is input into the gating network to generate pixel-level weights ω corresponding to each modality. i The multimodal density map is weighted using this weight, and the fused output is the final population probability density map; weight ω i It adapts to changes in the input gating network features, thereby achieving adaptive fusion of multimodal features.

[0030] 104: After obtaining the fused multimodal population probability density map, the number of people is obtained by integrating the density map; in addition, during the training process, the predicted probability density map will be compared with the ground truth probability density map to calculate the MSE loss, which is used to update the model parameters.

[0031] In summary, this method, consisting of a feature extraction module, a channel dynamic transfer module, and a decision-level adaptive fusion module, is a multimodal dynamic feature transfer crowd counting method. It can achieve interactive transfer between different modal features, thereby improving the counting effect of each modality and the overall system. It avoids the disadvantage of using only a single modality image for counting under poor imaging conditions and better utilizes multimodal features to improve crowd counting performance.

[0032] Example 2

[0033] The scheme in Example 1 will be further described below with specific examples and calculation formulas:

[0034] I. Data Preparation

[0035] The effectiveness of the proposed method was verified on the RGBT-CC and Drone-RGBT datasets in this embodiment of the invention.

[0036] RGBT-CC is a large-scale RGB-T population statistics dataset. This dataset includes various social scenes (shopping malls, streets, train stations, etc.), containing 2030 pairs of manually labeled RGB-T images. The dataset samples include both bright and dark scenes, with a total of 138,389 pedestrian markers, averaging 68 people per image. 1030 pairs were used for training, 200 pairs for validation, and 800 pairs for testing. The dataset's ground truth was generated using a geometrically adaptive Gaussian kernel.

[0037] The Drone-RGBT dataset is a multimodal dataset collected by drones, and is the first drone crowd counting dataset to combine RGB and T data. It contains 3600 image pairs taken at different locations, with an average of 48.8 people per image. Compared to other crowd counting datasets, this dataset exhibits greater diversity in lighting, viewpoint, and background. Furthermore, data augmentation was performed on the Drone-RGBT dataset before use, and annotation accuracy was improved through manual supplementation. 2285 pairs were used for training, and 1395 pairs were used for testing. The dataset uses a geometrically adaptive Gaussian kernel to generate the ground truth.

[0038] II. Multimodal Dynamic Feature Transfer Crowd Counting Network Structure

[0039] The multimodal dynamic feature transfer crowd counting network in this embodiment of the invention, such as... Figure 1 As shown, it includes three modules: a feature extraction network with two branches, a channel dynamic transfer module, and a decision-level adaptive fusion module.

[0040] In this embodiment of the invention, the counting network uses the commonly used convolutional neural network ResNet101 as the front-end network architecture. A dilated convolutional layer is added on the basis of ResNet101 as the back-end network. The final regression layer directly outputs a density map with an input size of 1 / 8. The output results of the two branches are weighted and fused to obtain the final density map.

[0041] To better utilize the information from the RGB and T modalities, this embodiment of the invention adds a channel dynamic migration module after Conv2_3, Conv3_4, Conv4_23, and Conv5_3 in the dual-branch backbone counting network. In this module, the interaction object selection submodule first selects the objects to be interacted with based on the distribution differences between corresponding channels in the modalities. Then, the interaction direction determination submodule determines the direction for further inter-modal interaction based on the distribution differences between the candidate interaction channels within a modality and the overall distribution. After the output of the dilated convolutional layer, a weighted exchange module performs weighted fusion of the outputs from the two branches. For example, this embodiment uses a gated network to assign specific weights to the two modalities based on the corresponding input data, and then weighted fuses the feature maps output from the regression layers of the two counting branches to obtain the final density map.

[0042] The aforementioned interaction object selection submodule is as follows:

[0043] After normalization, the Batch Normalization (BN) layer in the network uses the learnable parameters γ and β along the channel dimension to establish the distribution corresponding to each channel, which is the basic object for calculating the distribution differences. The distance metric between the two distributions represented by the corresponding channels of the visible light and infrared modes is as follows:

[0044]

[0045] Where P1 and P2 are the distributions corresponding to the two modes, and Π(P1,P2) is the set of all possible joint distributions of P1 and P2. Select n groups (n is a hyperparameter) with large distribution differences as the objects to be interacted with in the modality.

[0046] The interaction direction determination submodule is as follows:

[0047] After the interaction object selection submodule selects the channel objects to be exchanged, this submodule determines the direction of inter-modal channel-level information interaction. Within each modality, the PoE distribution of all channels is taken as the average distribution of that channel, where the PoE distribution is expressed as:

[0048] P avg =N(μ) k ,σ k )

[0049]

[0050]

[0051] μ0 and σ0 are the mean and variance of the standard normal distribution, respectively.

[0052] The distance metric between the distribution of channels to be interacted in each modality and the average distribution of that modality is as follows:

[0053]

[0054] Among them, Π(P,P) avg ) are P1 and P avg The set of all possible joint distributions is used to compare the distance between two sets of distributions. The mode that differs more from the average distribution will act as the receiver in the interaction, while the corresponding channel of the other mode will act as the giver.

[0055] The decision-level adaptive fusion module is as follows:

[0056] The multimodal feature extraction results are input into the gating network, and specific weights are assigned to the two modalities respectively:

[0057]

[0058]

[0059] in, and These are the outputs of the back-end network in the two-stream feature extraction network.

[0060] The feature maps output from the two counting branch regression layers are weighted and fused:

[0061] F=ω RGB F RGB +ω T F T

[0062] Among them, F RGB and F T These are the feature maps output by the two counting branches of the regression layer.

[0063] By performing pixel-level weighting on the multimodal feature extraction results and adaptively fusing information from different modalities, the final density map is obtained.

[0064] III. Evaluation Indicators and Agreements

[0065] To compute per-pixel density at each location in the image, a density map estimation task is performed while preserving spatial information about crowd distribution. This method uses mean absolute error (MAE) and mean squared error (RMSE) to evaluate performance, namely:

[0066]

[0067]

[0068] Where N is the number of images, Ci For the estimated count of the test image, The actual count for the corresponding image annotations.

[0069] The GMAE calculation for level l is as follows:

[0070]

[0071] in, and It is the estimated count of the j-th region of the i-th image and the actual count of the corresponding image annotation.

[0072] GMAE is used to evaluate the performance of different regions. GMAE(0) is equivalent to MAE. Specifically, the given image is divided into 4... l Non-overlapping regions were identified, and the counting error for each region was measured separately.

[0073] IV. Details on Model Usage

[0074] 1. Data Augmentation: Due to limited computing resources, a strategy of randomly flipping and cropping training images was adopted to increase the diversity of training data. For images larger than 680×640, the image size was first adjusted to be smaller than 680×640. Furthermore, manual annotation was used to improve the accuracy of the annotations.

[0075] 2. Model optimization:

[0076] This method sets the batch size N to 4 during training, and then uses the Adam optimization algorithm, achieving a batch size of 10 in the first 10 epochs. -6 The network was trained at a learning rate of 10^60 ... -5 The learning rate is used to train the network.

[0077] This method uses the geometrically adaptive Gaussian kernel method to generate the ground truth density map.

[0078] 3. Crowd Counting:

[0079] The density map of the two-branch fusion network is obtained through the crowd counting network in this method. This density map represents the probability of people appearing in the image and contains both spatial distribution features and population information. The number of people in the corresponding image or image module is obtained by integrating and summing the pixel density values ​​in the crowd density map. The mean squared error is used to measure the difference between the estimated density map and the true density map during network model training.

[0080] The embodiments of the present invention have the following three key inventive points:

[0081] I. A multimodal feature fusion crowd counting network is proposed.

[0082] Technical Results: This network enables interactive transfer between different modal features, thereby improving the counting performance of each modality. This multimodal feature fusion crowd counting network avoids the disadvantage of using only a single modality image for counting under poor imaging conditions, and better utilizes multimodal features to improve crowd counting performance.

[0083] II. Proposing a multimodal interaction mechanism based on distribution uncertainty

[0084] Technical effect: This mechanism utilizes a simple modular design and the parameters of the network structure itself to provide a more robust and flexible basis for interaction selection at the channel level, thereby guiding the interaction migration between different modalities. This mechanism can greatly improve the multimodal interaction counting effect.

[0085] III. An adaptive decision-level fusion strategy is proposed.

[0086] Technical benefits: By maximizing the integration of effective results from both modalities, this approach helps obtain more reliable results under unpredictable multimodal imaging conditions. Experimental validation was conducted on the two-light datasets RGBTCC and DroneRGBT.

[0087] In summary, this invention provides a multimodal dynamic feature transfer crowd counting method based on distribution uncertainty, designs a multimodal interaction mechanism based on distribution uncertainty, enables dynamic interaction and sharing between different modalities in the channel dimension during feature extraction, improves the feature extraction effect of each modality through bidirectional information transfer, and designs a decision-level adaptive fusion strategy, enabling the training model to overcome the influence of unpredictable imaging conditions and generate a more reliable density map.

[0088] Example 3

[0089] The method proposed in this invention is compared with several crowd counting methods on the RGBT-CC and Drone-RGBT datasets. For the unimodal crowd counting methods MCNN, SANet, CSRNet, BL, SASNet, and MAN, the "early fusion" results of RGB and T are used as network input; and compared with the multimodal methods CSRNet+IADM and BL+IADM.

[0090] The experimental results on the RGBT-CC dataset are shown in Table 1. All experimental metrics, GAME(0), GAME(1), GAME(2), GAME(3), and RMSE, are 13.12, 17.55, 22.11, 27.15, and 22.34, respectively, all outperforming other comparative methods. Compared with unimodal crowd counting methods, the methods in this embodiment of the invention achieve better evaluation results. Compared with multimodal crowd counting methods, this method improves the evaluation metric GMAE by 4.82 and 2.49, respectively, and the evaluation metric RMSE by 8.57 and 5.84, respectively. Therefore, it can be demonstrated that the methods in this embodiment of the invention can dynamically combine the advantages of multimodal data and better utilize the complementary characteristics of visible light images and thermal infrared images.

[0091] The experimental results on the Drone-RGBT dataset are shown in Table 2. All experimental metrics, GAME(0), GAME(1), GAME(2), GAME(3), and RMSE, are 9.14, 10.49, 12.92, and 14.69, respectively, all outperforming other comparative methods. Compared to unimodal crowd counting methods, the methods in this embodiment achieve better evaluation results. Compared to multimodal crowd counting methods, this method improves upon the GMAE metric by 1.65 and 0.69, respectively, and improves upon the RMSE metric by 2.68 and 1.29, respectively. Therefore, this demonstrates that the methods in this embodiment can be applied to RGB-T datasets under various scenarios and perspectives, further validating the effectiveness of the method.

[0092] The experimental results of this invention are shown in Table 3. These results demonstrate the performance of three variants of the proposed method on the RGBT-CC dataset: DFTNet(w / o all), DFTNet(w / o fus), and DFTNet(w / otran). DFTNet(w / o fus) represents a network without decision-level adaptive fusion, where the two subnets have the same weights and are fused later. DFTNet(w / o tran) represents a network without a channel dynamic transfer module. DFTNet(w / o all) represents a network without both a channel dynamic transfer module and a decision-level adaptive fusion module. All variants were trained on the training set and tested on the test set. The training steps and other parameters were identical across the different experiments, and the evaluation was the same. As shown in Table 3, this method achieved better results than other variants, validating that the channel dynamic interaction module and the decision-level adaptive fusion module can significantly improve the performance of the density map estimation task.

[0093] Table 1

[0094]

[0095]

[0096] Table 2

[0097] method GMAE(0) GMAE(1) GMAE(2) GMAE(3) MSE MCNN 15.56 16.68 18.33 19.91 22.49 SANet 16.42 17.52 19.62 21.81 22.57 CSRNet 11.68 13.73 16.39 18.83 16.69 CSRNet+IADM 10.79 12.9 15.33 17.19 17.24 BL 11.14 13.74 17.57 22.54 16.56 BL+IADM 10.47 12.25 14.77 18.57 16.44 SASNet 12.86 14.12 17.07 18.89 18.96 MAN 11.35 13.56 16.21 17.98 17.33 DFTNet 9.14 10.49 12.92 14.69 14.56

[0098] Table 3

[0099]

[0100]

[0101] Example 4

[0102] A multimodal dynamic feature migration crowd counting device based on distribution uncertainty, see [link to relevant documentation]. Figure 3 The device includes a processor and a memory, wherein the memory stores program instructions, and the processor invokes the program instructions stored in the memory to cause the device to perform the following method steps in Embodiment 1:

[0103] The multimodal dynamic feature transfer counting network uses ResNet101 residual network as the backbone, extracts multimodal features with a two-stream network structure, and generates a population probability density map.

[0104] In the process of multimodal feature extraction, feature information exchange between modalities is carried out based on the channel dynamic interaction mechanism based on distribution uncertainty;

[0105] Obtain the probability density map of the output of the dual-stream feature extraction network, and realize the adaptive fusion of multimodal output results through the decision-level adaptive fusion module;

[0106] After obtaining the fused multimodal population probability density map, the population number is obtained by integrating the density map.

[0107] This also includes: during training, the predicted probability density map is compared with the ground truth probability density map to calculate the MSE loss, which is used to update the model parameters.

[0108] Furthermore, the multimodal dynamic feature transfer crowd counting network includes a feature extraction network with two branches, a channel dynamic transfer module, and a decision-level adaptive fusion module.

[0109] The channel dynamic migration module is as follows:

[0110] The interaction object selection submodule selects the objects to be interacted with based on the distribution differences between corresponding channels in the modal; the interaction direction determination submodule determines the direction of interaction between modals based on the distribution differences between the candidate interaction channels within a modal and the overall distribution.

[0111] Furthermore, the interaction object selection submodule is as follows:

[0112]

[0113] Where P1 and P2 are the distributions corresponding to the two modes, and Π(P1,P2) is the selection of n groups with large distribution differences as the objects to be interacted with in the modality.

[0114] Furthermore, the interaction direction determination submodule is as follows:

[0115]

[0116] Where P(P,P) avg ) are P1 and P avg The set of all possible joint distributions that can be combined.

[0117] The decision-level adaptive fusion module is as follows:

[0118] The multimodal feature extraction results are input into the gating network, and specific weights are assigned to the two modalities respectively:

[0119]

[0120]

[0121] in, and These are the outputs of the back-end network in the two-stream feature extraction network;

[0122] The feature maps output from the two counting branch regression layers are weighted and fused:

[0123] F=ω RGB F RGB +ω T F T

[0124] Among them, F RGB and F T These are the feature maps output by the two counting branches of the regression layer.

[0125] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0126] The execution entities of the processor 1 and memory 2 mentioned above can be devices with computing functions such as computers, microcontrollers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution entity and can select it according to the needs of actual application.

[0127] Data signals are transmitted between memory 2 and processor 1 via bus 3, which will not be described in detail in this embodiment of the invention.

[0128] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the storage medium including a stored program, which, when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiments.

[0129] The computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc.

[0130] It should be noted that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be repeated here.

[0131] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0132] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.

[0133] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0134] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0136] References:

[0137] [1]Huang X,Mallya A,Wang TC,et al.Multimodal Conditional ImageSynthesis with Product-of-Experts GANs[J].2021.

[0138] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0139] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0140] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for counting multimodal dynamic feature migration populations based on distribution uncertainty, characterized in that, The method includes: The multimodal dynamic feature transfer counting network uses ResNet101 residual network as the backbone, extracts multimodal features with a two-stream network structure, and generates a population probability density map. In the process of multimodal feature extraction, feature information exchange between modalities is carried out based on the channel dynamic interaction mechanism based on distribution uncertainty; Obtain the probability density map of the output of the dual-stream feature extraction network, and realize the adaptive fusion of multimodal output results through the decision-level adaptive fusion module; After obtaining the fused multimodal population probability density map, the population number is obtained by integrating the density map. The multimodal dynamic feature transfer crowd counting network includes a two-branch feature extraction network, a channel dynamic transfer module, and a decision-level adaptive fusion module; The channel dynamic migration module is: The interaction object selection submodule selects the objects to be interacted with based on the distribution differences between corresponding channels in different modalities; the interaction direction determination submodule determines the direction of interaction between modalities based on the distribution differences between the candidate interaction channels within a modal and the overall distribution. The interaction object selection submodule is as follows: ； Wherein, P1 and P2 are the distributions corresponding to the two modes, respectively. The method involves selecting n groups with large distribution differences as the objects to be subjected to modal interaction. The interaction direction determination submodule is: ； in, It is P1 and P avg The set of all possible joint distributions that can be combined.

2. The method for counting multimodal dynamic feature migration populations based on distribution uncertainty according to claim 1, characterized in that, The method further includes: during training, the predicted probability density map is compared with the ground truth probability density map to calculate the MSE loss, which is used to update the model parameters.

3. The method for counting multimodal dynamic feature migration populations based on distribution uncertainty according to claim 1, characterized in that, The decision-level adaptive fusion module is: The multimodal feature extraction results are input into the gating network, and specific weights are assigned to the two modalities respectively: ； in, and These are the outputs of the back-end network in the two-stream feature extraction network; The feature maps output from the two counting branch regression layers are weighted and fused: ； in, and These are the feature maps output by the two counting branches of the regression layer.

4. A multimodal dynamic feature migration crowd counting device based on distribution uncertainty, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor invoking the program instructions stored in the memory to cause the device to perform the method according to any one of claims 1-3.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-3.

Images