Smart education scene crowd counting method based on meta-learning and multi-modal large model

Abstract

The invention requests to protect a wisdom education scene crowd counting method based on meta-learning and a multi-modal large model. The method comprises the following steps: positioning individual positions through a point regression mode by branches from top to bottom, and outputting a first people number predicted value; a large language model is called through a space prior analysis module from bottom to top to generate regionalized semantic description, vision-language feature alignment and fusion are achieved through a regionalized multi-head cross-attention module, and then a second people number predicted value is obtained through density map regression; the double-branch features are further fused to form a third path, and a third people number predicted value is output; and finally, adaptive weighting is carried out on the three-path prediction result through a learnable weight, and a final crowd counting result is obtained. In the model training stage, a meta-learning strategy is adopted, and supervised optimization is carried out through a multi-task loss function in combination with point regression loss, counting loss and optimal transmission loss, so that the model has rapid scene adaptation capability under the condition of few samples.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, specifically a method for counting people in smart education scenarios based on meta-learning and multimodal large models. Background Technology

[0002] Crowd counting, a crucial branch of computer vision, has been widely applied in fields such as smart education, public safety, and commercial traffic monitoring. Currently, mainstream methods primarily follow two technical approaches: one focuses on directly regressing density maps or count values ​​from image pixels, improving performance through deeper convolutional neural networks or the introduction of multi-scale modules; the other attempts to directly obtain individual location information through detection or localization. However, in the specific scenario of smart education, existing methods face multiple challenges.

[0003] First, the severe scale variation problem caused by differences in camera placement, angle, and height in surveillance footage has not been effectively resolved. Students' heads appear larger and more defined near the camera, while those at a distance are extremely small and blurry, sometimes occupying only a few pixels. This makes it difficult for a single model to accurately capture all targets in the entire image simultaneously. While density map regression methods are robust to dense crowds, they cannot provide precise individual location information; and existing localization methods are prone to false negatives or missed detections in highly crowded scenarios. Second, existing methods have significant shortcomings in feature representation. Campus scenes possess rich semantic information, such as evenly distributed students in classrooms, dynamically changing crowds on the playground, and multiple high-density queuing areas in the cafeteria. However, traditional methods, when fusing multimodal information, typically employ simple splicing or addition operations, lacking explicit modeling of image spatial structure and perspective principles. This coarse-grained fusion approach struggles to achieve precise alignment between semantic descriptions and corresponding image regions, failing to fully utilize the prior knowledge of structured regions generated by LLM. Third, regarding the design of supervisory signals, existing crowd counting methods generally rely on training with "pseudo-true" density maps generated by smoothing point annotations with Gaussian kernels. The selection of Gaussian kernel parameters lacks theoretical basis and is subjective, failing to accurately reflect the uneven distribution of crowds caused by perspective effects and occlusion in real-world scenarios. Furthermore, density map-based regression methods and point-based localization methods each have inherent biases and lack effective coordination mechanisms.

[0004] In the field of few-shot cross-scene adaptation, current research largely focuses on directly fine-tuning pre-trained models or employing metric-based learning methods. However, these methods typically rely on simple pixel-level reconstruction losses, failing to effectively constrain the global structural consistency between predicted results and the true distribution. Consequently, when faced with new classrooms or layouts, the model's generalization ability significantly declines, making rapid adaptation difficult. Although Optimal Transport (OT) theory has been introduced to compensate for insufficient supervision at the distribution level, it has not yet been deeply integrated with few-shot learning mechanisms, nor has it addressed the issue of multi-method collaborative optimization.

[0005] Therefore, this invention proposes a dual-branch fusion architecture, which provides accurate individual localization through a top-down point regression branch, provides stable density estimation by combining a bottom-up multimodal fusion branch, and finally fully utilizes the complementary advantages of different methods through a weighted fusion mechanism of three-path prediction to solve the cross-perspective and cross-layout generalization problem faced in smart education scenarios.

[0006] A search revealed application publication number CN118470630A, which discloses a cross-domain small-sample crowd counting method based on meta-learning, comprising the following steps: Step S1: Sampling a synthetic crowd counting dataset and a real-world crowd counting dataset to construct a meta-training set and a meta-test set; Step S2: Constructing a scene adaptation module that dynamically adjusts the network based on scene context information; Step S3: Constructing a crowd counting model; Step S4: Designing a training strategy based on meta-learning; Step S5: Using the model obtained in Step S4, testing it on the meta-test set using a regularization fine-tuning method based on knowledge distillation. This invention significantly reduces the cost of data collection and annotation; the use of a regularization fine-tuning method based on knowledge distillation solves the overfitting problem; and it also introduces a scene adaptation module that adjusts the network based on scene context information to improve the model's generalization ability in cross-domain situations.

[0007] The existing technology represented by the published patent CN118470630A has two main drawbacks: First, the model relies on single visual features and density map regression, lacking the utilization of high-level semantic information of the scene, resulting in insufficient understanding and generalization ability in complex and ever-changing real-world scenarios; second, its scene adaptation module and supervision signal are relatively simple, failing to effectively coordinate the advantages of localization and counting methods, and its over-reliance on Gaussian kernel-generated density maps introduces inherent biases. To address these issues, this invention proposes a creative solution: by introducing a large-scale language model to generate regionalized semantic descriptions, a multimodal understanding framework integrating vision and language is constructed; simultaneously, an innovative dual-branch architecture combining top-down point regression and bottom-up density map regression is adopted, and a meta-learning strategy is used to collaboratively optimize point regression, counting, and optimal transmission loss, thereby significantly improving the model's counting stability and cross-scene generalization ability under few-sample conditions while retaining accurate localization capabilities. Summary of the Invention

[0008] To address the shortcomings of existing crowd counting models in terms of scale adaptability, cross-scenario generalization ability, and excessive reliance on Gaussian kernel smoothing for supervision signals, this invention proposes a crowd counting method for smart education scenarios based on a dual-branch architecture and a multimodal large model. This method achieves high-precision and robust crowd counting by combining the advantages of point regression and density map regression through the collaborative processing of top-down and bottom-up dual-branch approaches.

[0009] The technical solution of the present invention is as follows:

[0010] A method for counting people in smart education scenarios based on meta-learning and multimodal large models includes the following steps:

[0011] The process involves: acquiring an image to be detected; inputting the image into a pre-trained dual-branch crowd counting model; performing feature extraction and point regression on the image through the top-down branch of the dual-branch crowd counting model to obtain a first predicted number of people; performing multimodal feature fusion and density map regression on the image through the bottom-up branch of the dual-branch crowd counting model to obtain a second predicted number of people; fusing the features from the top-down and bottom-up branches through a feature fusion module, and obtaining a third predicted number of people based on the fused features; and finally, weighted summing of the first, second, and third predicted number of people through a weighted fusion module to obtain the final crowd counting result.

[0012] Furthermore, the step of obtaining the first predicted number of people through the top-down branch includes: extracting the second, third, and fourth layer features of the input image using a VGG16 backbone network; downsampling the second layer features, upsampling the fourth layer features, and concatenating and fusing them with the third layer features to form top-down branch features; predicting the two-dimensional coordinates and confidence scores of head points using a point regression head based on the top-down branch features; and associating the predicted points with the real labeled points using a greedy matching algorithm based on the spatial distance between the predicted points and the real labeled points, and obtaining the first predicted number of people based on the number of successfully matched points.

[0013] Furthermore, the step of obtaining the second predicted number of people through the bottom-up branch includes: using a spatial prior parsing module, a large language model is invoked based on a predefined prompt word template to adaptively parse the input image into spatial regions such as far-field, mid-field, and near-field, and generate structured semantic descriptions corresponding to the distribution state of the crowd in each region; the semantic descriptions are processed by a text encoder of a vision-language model to generate regionalized semantic features; the visual features of the image are extracted by an image encoder of a vision-language model; and the regionalized semantic features and the image visual features are input into a region-aware multi-head cross-attention (R-MCA) module for fusion. This module uses the regionalized semantic features generated in the aforementioned steps as a query matrix, and the visual features extracted by the image encoder as the key matrix and value matrix. In the calculation of the attention weights, a learnable positional bias based on the image normalized vertical coordinates and generated by a multilayer perceptron is introduced; based on the fused features, a density map is predicted by a density regression module, and the density map is spatially integrated to obtain the second predicted number of people.

[0014] Furthermore, the weighted fusion module uses learnable weight parameters for weighted summation, and these learnable weight parameters are automatically optimized through gradient descent during the training process of the dual-branch crowd counting model.

[0015] Furthermore, the method also includes a meta-learning training approach for the crowd counting model, comprising: acquiring crowd counting datasets from multiple scenarios and constructing a meta-training task set based on these datasets, wherein each task contains a support set and a query set for a scenario; performing an outer loop meta-optimization process: sampling a batch of tasks from the meta-training task set, and for each task: on the support set, performing one or more steps of gradient descent on the current principal parameters of the model based on a multi-task loss function to obtain task adaptive parameters; on the query set, calculating the loss using the task adaptive parameters; updating the principal parameters of the model based on the loss calculated on the query set; and iteratively executing the outer loop meta-optimization process to converge the principal parameters of the model, thereby obtaining a meta-initial model with rapid cross-scenario adaptability.

[0016] Furthermore, the specific steps for fusing the regionalized semantic features and image visual features into the regionalized multi-head attention module include:

[0017] S21: Receive image patch features X extracted by the visual encoder and regionalized text features extracted by the text encoder. ; This represents the feature in the feature space of the text description generated by LLM corresponding to each image patch.

[0018] S22: Transform the regionalized text features T into a query matrix Q through a linear projection layer; transform the image region features X into a key matrix K and a value matrix V through linear projection layers respectively;

[0019] S23: For each image patch, based on its normalized vertical coordinates... The bias value corresponding to each attention head h is generated through an independent multilayer perceptron (MLP) network. Together they form the position offset matrix ;

[0020] S24: Add the positional bias to the attention scoring function to calculate the interaction weights between each region's text token and all image patches:

[0021]

[0022] Each region's text feature vector serves as an independent query, used to focus on the corresponding spatial region in the image; is the dimension of each key vector in the key matrix K, used here as a scaling factor. It is determined by the positional bias of all attention heads and all image patches. The resulting position offset matrix.

[0023] S25: The weighted features of the attention output are added to the original image Patch features X through a residual connection to obtain the fused cross-modal feature representation. ;

[0024] .

[0025] Furthermore, the multi-task loss function includes point regression loss, counting loss, and optimal transmission loss.

[0026] Furthermore, when updating the model parameters on the support set using gradients, the MAML algorithm is employed, and the optimized parameters include at least one of the following: learnable semantic cue vector, point regression head weights, density regression head weights, and weighted fusion weights.

[0027] An electronic device includes: one or more processors; a memory; and one or more computer programs; wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and the one or more computer programs include instructions for performing the methods as described in any one of the claims.

[0028] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the crowd counting method as described in any one of the claims.

[0029] The advantages and beneficial effects of this invention are as follows:

[0030] This invention innovatively proposes a dual-branch collaborative processing architecture, organically combining point regression localization mechanism with multimodal feature fusion mechanism, achieving breakthrough progress in crowd counting tasks. Through the top-down branch's point regression method, the model can directly output the precise location coordinates and confidence scores of individuals, effectively improving localization accuracy and individual recognition accuracy. Simultaneously, the bottom-up branch, through deep collaboration between LLM and VLM, constructs a spatial prior parsing module, which can automatically generate high-quality scene semantic descriptions with spatial hierarchical structure, overcoming the subjectivity and limitations of manually constructed prompts. Furthermore, through a learnable semantic prompting mechanism and a regionalized multi-head cross-attention fusion module (R-MCA), deep scene prior knowledge is dynamically and accurately embedded into the model. The R-MCA module ensures a strong correlation between semantic priors and image spatial location, enabling the model to not only utilize image pixel features but also deeply fuse semantic information in the spatial dimension, achieving effective alignment and fusion of text prompts and image information. Regarding the supervision mechanism, this invention innovatively combines point regression loss, counting loss, and optimal transmission loss to construct a multi-task supervision framework. This design abandons the reliance on traditional Gaussian kernel diffusion methods, accurately measuring the consistency between the predicted distribution and the actual point distribution through optimal transport theory. Simultaneously, it utilizes point regression loss to ensure the accuracy of individual location, significantly enhancing the model's robustness and generalization ability. In particular, through a weighted fusion mechanism of three-path prediction, this invention fully leverages the location accuracy advantage of point regression and the stability advantage of density map regression, forming a good complementary effect. Dual-branch feature fusion further enhances the model's representational ability, enabling it to simultaneously capture global semantic information and local detailed features, demonstrating stronger adaptability in dealing with challenges such as scale changes, perspective differences, and layout diversity. Through these innovative designs, this invention achieves rapid adaptation and high-precision crowd counting in smart education scenarios, ensuring both the accuracy of individual location and the stability of overall counting. This provides more reliable and comprehensive technical support for applications such as classroom personnel monitoring, playground activity management, meeting attendance analysis, and cafeteria crowd management. Attached Figure Description

[0031] Figure 1 This is a framework diagram of a preferred embodiment of the present invention, which utilizes meta-learning optimization to provide a dual-branch crowd counting method based on LLM and VLM;

[0032] Figure 2 This is a structural diagram of the bottom-up feature extraction module proposed in this invention;

[0033] Figure 3 This is a structural diagram of the top-down feature extraction module proposed in this invention;

[0034] Figure 4This is a framework diagram of the regionalized multi-head cross-attention fusion module of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.

[0036] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0037] The method described in this invention employs a dual-branch processing framework and specifically includes the following steps:

[0038] First, a top-down branch is constructed using a point regression-based localization and counting mechanism. Multi-layered visual features of the input image are extracted using a VGG16 backbone network. The second-layer features are downsampled, the third-layer features are kept at their original scale, and the fourth-layer features are upsampled. These features are then concatenated and fused to form the top-down branch features. Based on these features, the two-dimensional coordinates of head points and their corresponding confidence scores are directly predicted using a point regression head. A greedy matching algorithm is then used to associate the predicted points with the ground truth labeled points. The predicted number of people in the top-down branch is obtained based on the number of successfully matched points.

[0039] Simultaneously, a bottom-up approach is constructed to retain and improve the multimodal feature fusion mechanism. The input image is fed into a spatial prior parsing module, which, based on a Large Language Model (LLM), divides the image into multiple patches according to a predefined input template (given an LLM prompt strategy: you are an image analysis AI model specifically designed to adaptively divide an image into multiple patches based on the spatial structure of the image content. The description of each patch should be based on its specific content, generating a natural and detailed text description involving the distribution, activities, and relationships of objects in the image. You need to generate a concise and accurate description of each region, in the following format: Region i: "Describe the content of the region", for example: "The area below the image is the podium area, there is no one there" or "There are several students chasing and playing at the back of the classroom"). Subsequently, the structured semantic description is fused with the learnable semantic prompt using a Visual-Language Model (VLM) text encoder to generate regionalized semantic features; simultaneously, the image visual features are extracted using the VLM image encoder. Next, cross-modal feature alignment is achieved through a regionalized multi-head cross-attention module (R-MCA). This module uses semantic features as the query and image patch features as the key / value pair, and introduces a learnable positional bias based on the image's vertical coordinate. This allows the semantic descriptions of different regions to be accurately aligned to their corresponding positions in the image, generating fused features that have undergone spatial-semantic modulation. This feature is then input into a density regression module to predict the density map, and the bottom-up branch population prediction is obtained through integration. Further, the top-down branch features are concatenated with the bottom-up branch features to form an enhanced core feature, which is then input into an independent density regression module to predict a third density map. The fused branch population prediction is then obtained through integration. Finally, the three path prediction results are weighted and fused using learnable weights.

[0040] The training process of the model adopts a meta-learning paradigm, constructing a meta-training task set containing N scenarios, each task representing a different scenario, and randomly selecting tasks from the task distribution for training. In the inner loop, the support set is used to update the task adaptive parameters through gradients; in the outer loop, the query set is used to calculate the loss and the gradients are used to update the model's master parameters (initial parameters). The loss function includes the point regression loss of the top-down branch (including coordinate regression loss and confidence classification loss), the counting loss of the bottom-up branch and the fusion branch, and the optimal transfer loss. These losses work together through a multi-task supervision mechanism to ensure the model's rapid adaptation ability from the source domain to the target domain under conditions of few samples. Through the above dual-branch architecture and three-path prediction fusion, this invention retains the precise localization advantage of the point regression method, leverages the stability of density map regression in different scenarios, and fully utilizes semantic prior information through multimodal feature fusion, ultimately achieving high-precision crowd counting across perspectives and scenarios in smart education scenarios.

[0041] This invention provides a dual-branch crowd counting method based on meta-learning optimization. Its core lies in constructing a collaborative processing framework that integrates point regression and multimodal density map regression. The method mainly includes three core stages: model construction, meta-training, and rapid adaptation. First, a dual-branch crowd counting model is constructed: the top-down branch directly locates the individual positions in the image through point regression, outputting the first predicted number of people; the bottom-up branch uses a spatial prior parsing module to call a large language model to generate regionalized semantic descriptions, and utilizes a visual-language model and a regionalized multi-head attention module to align and fuse visual-semantic features, then outputs the second predicted number of people through density map regression; finally, the prediction results of the two branches and the fusion path are fused using learnable weights to obtain the final count. In the model training stage, a meta-learning strategy is adopted, enabling the model to learn on a large number of heterogeneous scene tasks, gaining rapid cross-scene adaptability. When applied to a new target scene, only a small set of support images and their point annotations are needed; the model parameters are quickly adapted through the meta-learning mechanism to achieve high-precision crowd counting for the query image in that scene.

[0042] The dual-branch crowd counting meta-learning model includes the following core modules: a top-down branch VGG16 feature extraction network, a point regression head, and a greedy matching module; a bottom-up branch pre-trained CLIP image encoder and text encoder, a learnable semantic prompting module, and a regionalized multi-head cross-attention fusion module; a feature fusion module; and a three-path prediction weighted fusion module.

[0043] The training of the crowd counting meta-learning model includes the following steps: Meta-learning tasks are constructed by sampling from multiple source scene datasets, each task containing a support set and a query set; the model's master parameters are optimized in the outer loop, including learnable semantic cue vectors, point regression head weights, and density regression head weights; for each task, the model is updated with one or more gradients using its support set data in the inner loop, with point regression loss, counting loss, and optimal transport loss serving as multi-task supervision signals to achieve rapid adaptation of task-specific parameters; the performance of the adapted model is evaluated on the query set, and the gradient of the loss with respect to the master parameters is calculated based on this performance to update the master parameters; the above process is repeated until the master parameters converge, ultimately obtaining meta-initial parameters with rapid cross-scene adaptability.

[0044] In this embodiment, as Figures 1-4As shown, the dual-branch crowd counting meta-learning model framework demonstrates the core implementation of this invention. In this framework, the meta-learning optimization strategy and the dual-branch processing mechanism work together, with the specific process as follows: In the top-down branch, the VGG16 backbone network extracts multi-level features from the input image, which are then downsampled, upsampled, and concatenated to form top-down branch features. Point coordinates and confidence scores are predicted using a point regression head, and then the predicted number of people in the top-down branch is obtained through a greedy algorithm. In the bottom-up branch, the spatial prior parsing module (based on LLM) automatically generates structured semantic descriptions corresponding to different regions based on the input image. This descriptive text is processed by the CLIP text encoder. After processing, semantic features are generated by fusing with learnable semantic cue vectors. Simultaneously, the CLIP image encoder extracts visual features from the image. These two features are then deeply fused through a regionalized multi-head cross-attention fusion module to form a semantically modulated visual representation. A density regression head then predicts a density map and integrates it to obtain bottom-up branch population predictions. Further, the dual-branch features are concatenated and fused to form core features. A third density regression head then predicts a third density map and integrates it to obtain the fused branch population predictions. Finally, learnable weights are used to weight and fuse the three-path prediction results to obtain the final crowd counting result. A multi-task loss function is used to calculate the point regression loss, counting loss, and optimal transmission loss, serving as the core supervision signal for model inner loop adaptation. Through collaborative optimization of the inner and outer loops in meta-learning, the model ultimately gains the ability to quickly understand the semantic information of new scenes and accurately estimate the spatial distribution of the crowd.

[0045] This invention, based on a general meta-learning framework and a dual-branch processing architecture, is flexibly applicable to various educational management scenarios, such as classroom monitoring, library visitor statistics, and peak-hour traffic monitoring in cafeterias. By introducing a dual-branch complementary mechanism and three-path prediction fusion, this method demonstrates significant advantages in counting accuracy, positioning precision, and cross-scenario generalization ability.

[0046] A specific implementation method for a few-shot population counting method in educational scenarios based on meta-learning and a dual-branch architecture is described, including the following steps:

[0047] S1: Sample images and point annotations from multiple source population counting datasets to construct a meta-training task set. Each task contains a support set and a query set from N different scenarios, and each scenario contains K samples.

[0048] S2: Construct a dual-branch crowd counting meta-learning model, including a bottom-up branch pre-trained CLIP image encoder and text encoder, a spatial prior parsing module, a learnable semantic prompting module, a regionalized multi-head cross-attention fusion module, a feature fusion module, and a three-path prediction weighted fusion module.

[0049] S3: Design a multi-task learning and meta-learning optimization strategy suitable for the above model; the strategy includes complementary enhancement by fusing features from two branches, adapting task parameters with inner loop multi-task loss (point regression loss, counting loss and OT loss), and updating the model master parameters with outer loop meta-update.

[0050] S4: Using the meta-training task set constructed in S1, execute the strategy designed in S3 to perform meta-training on the dual-branch crowd counting meta-learning model, and optimize the model parameters until convergence.

[0051] S5: Obtain a small number of images and point annotations of the target new scene for meta-testing. Use the spatial prior parsing module to generate a scene description, input it into the trained dual-branch model for meta-testing, and obtain the optimal model parameters adapted to the current task through rapid adaptation. The adapted model can then perform high-precision crowd counting on the query images of the new scene.

[0052] S11: Collect image and point annotation data from multiple publicly available crowd counting datasets, including but not limited to MALL, ShanghaiTech Part_A, ShanghaiTech Part_B, UCF-QNRF, and NWPU-Crowd. Classify the scenes according to their visual features and crowd distribution characteristics, with each scene category considered a candidate task category.

[0053] S12: Randomly select N different scene categories from the candidate task categories to form an N-way meta-learning task, where the value of N ranges from 3 to 8, to ensure task diversity while maintaining training stability.

[0054] S13: In each selected scene category, randomly select K images with point annotations as the support set and retain the corresponding point annotation information, where K ranges from 1 to 10 to simulate few-shot learning conditions.

[0055] S14: Within the same scene category, additional M images are randomly selected as a query set, and their ground truth point labels are retained. The value of M ranges from 5 to 20 to ensure the statistical significance of the evaluation.

[0056] S15: Label and store the support set and query set separately to form a complete meta-training task sample pair, containing...

[0057] S16: Repeat steps S12-S15 until a meta-training task set containing hundreds to thousands of tasks is constructed for the meta-training phase of the model. The task set should cover different scene complexities, crowd densities, and perspective variations to ensure the model's generalization ability.

[0058] S17: In the meta-testing phase, a small number of support and query sets are constructed from the target educational scenario dataset (including proprietary datasets such as classroom monitoring, playground activities, cafeteria queuing, and meeting room attendance) using the same N-way K-shot sampling method. These sets serve as the target new scenario task set for meta-testing, enabling rapid model adaptation. The target scenario dataset is not involved in the meta-training process to verify the model's cross-scenario generalization ability.

[0059] In this embodiment, regionalized multi-head cross-attention (R-MCA) combined with learnable location mapping is used to achieve the fusion of text features and image features. The specific steps include:

[0060] S21: Receive image patch features X extracted by the visual encoder and regionalized text features extracted by the text encoder. .

[0061] S22: Transform the regionalized text features T into a Query matrix Q through a linear projection layer; transform the image region features X into a Key matrix K and a Value matrix V through linear projection layers respectively.

[0062] S23: For each image patch, based on its normalized vertical coordinates... The bias value corresponding to each attention head h is generated through an independent multilayer perceptron (MLP) network. Together they form the position offset matrix .

[0063] S24: Add the positional bias to the attention scoring function to calculate the interaction weights between each region's text token and all image patches:

[0064]

[0065] In this context, the text feature vector of each region serves as an independent query, used to focus on the corresponding spatial region in the image.

[0066] S25: Add the weighted features of the attention output to the original image Patch features X through a residual connection to obtain the fused cross-modal feature representation. .

[0067]

[0068] In this embodiment, meta-training of the model using CLIP as the graph front-end network includes:

[0069] In this embodiment, the meta-training of the dual-branch crowd counting model using CLIP as the graph front-end network includes the following steps:

[0070] S41: Sample a task from the meta-training task set and obtain its support set images and corresponding ground truth point annotation data.

[0071] S42: The images from the support set are input into a dual-branch architecture: Top-down branch: Input to the VGG16 backbone network to extract multi-level visual features, and after feature fusion, input to the point regression head to predict the coordinates and confidence of the head points. The first predicted number of people is obtained through greedy matching, and the point regression loss is calculated. The bottom-up approach involves inputting the image into a spatial prior analysis module to generate a structured region semantic description; processing the semantic description through a CLIP text encoder and fusing it with a learnable cue vector to obtain semantic conditional features; simultaneously extracting visual features through a CLIP image encoder; achieving cross-modal feature fusion via a regionalized multi-head cross-attention module; and predicting the density map through a density regression head and integrating it to obtain the second number prediction value.

[0072] S43: The dual-branch features are spliced ​​and fused through the feature fusion module to form an enhanced core feature. The independent density regression head is then used to predict the third density map, and the integral is used to obtain the predicted value of the third number of people.

[0073] S44: The weighted fusion module performs weighted fusion of the three path prediction results to obtain the final population count result.

[0074]

[0075] in These are learnable weight parameters, all initialized to 1 / 3, and optimized together with the model's main parameters during the outer loop of meta-learning. These represent the predicted number of people for the top-down branch, the bottom-up branch, and the feature fusion branch, respectively.

[0076] S45: Calculate the multi-task loss function, including point regression loss. Includes coordinate regression loss and confidence classification loss; counting loss. L1 norm loss between predicted and actual three-path counts; optimal transmission loss Wasserstein distance between the predicted density map and the true point distribution.

[0077] The total loss is:

[0078]

[0079] S46: Multi-task loss based on support set in the inner loop. Perform a gradient descent step on the task-specific parameters (including the learnable semantic cue vector, point regression head weights, density regression head weights, and weighted fusion weights) to obtain the task-adaptive parameters. :

[0080] S47: Use the adapted parameters Perform forward propagation on the query set for this task and calculate the query set loss. .

[0081] S48: Calculate the query set loss with respect to the original master parameters. gradient .

[0082] S49: Aggregate the query set loss gradients from multiple tasks and update the model's main parameters. :

[0083]

[0084] Complete one outer loop optimization.

[0085] S410: Repeat steps S41 to S49 until the main parameter is reached. Convergence yields meta-initial parameters capable of rapid adaptation across different scenarios.

[0086] In this embodiment, the meta-testing of the trained dual-branch model using CLIP as the graph front-end network includes the following steps:

[0087] S51: Obtain a small set of support images and their corresponding ground truth point annotation data for the new target education scenario.

[0088] S52: Supports a pre-trained dual-branch model with image input: the top-down branch extracts features using VGG16, predicts individual positions using a point regression head, and obtains the first number prediction value; the bottom-up branch generates scene semantic descriptions using LLM, extracts multimodal features using a CLIP encoder, fuses features across attention modules in a regionalized manner, and obtains the second number prediction value using a density regression head; the feature fusion branch fuses the features from both branches and obtains the third number prediction value using an independent regression head. The number of people obtained from the three branches is then weighted and fused: the final counting result is obtained by fusing the three path predictions.

[0089] S53: Calculate the multi-task loss on the support set:

[0090]

[0091] S54: Based on the inner loop objective function, fix the parameters of the backbone network of the model, and perform one or more steps of gradient descent only on the task-specific parameters (learnable cue vectors, regression head weights, fusion weights) to obtain task-specific parameters adapted to the new scenario. :

[0092]

[0093] S55: Using adapted parameters The query image for this new scene is forward-propagated to output high-precision crowd count results, individual location coordinates, and density map.

[0094] During the training phase, the dual-branch architecture and the meta-learning framework are jointly optimized end-to-end, and the final loss function is composed of point regression loss, counting loss, and optimal transmission loss.

[0095] Definition of count loss

[0096] make Describes the L1 norm of a vector. and These are the actual count and the predicted count. The goal of crowd counting is to... as close as possible Count loss Defined as the sum of the absolute differences between the predicted and actual values ​​of the three paths:

[0097]

[0098] in These represent the predicted number of people for the top-down branch, the bottom-up branch, and the feature fusion branch, respectively.

[0099] During the training phase, the fusion module performs end-to-end joint optimization with the entire network, and the final loss function is derived from the counting loss. With optimal transmission loss Together, they form a model to ensure that the model is consistent with the true value in terms of both counting accuracy and distribution pattern.

[0100]

[0101]

[0102] make Denotes the L1 norm of a vector, therefore and These are the actual counts and the density map predicted counts, respectively. The goal of the crowd count is to... as close as possible Count loss Defined as the absolute difference between the two.

[0103] Define the transmission cost matrix Its elements This indicates the position of a unit mass from the true distribution. Move to the position in the predicted distribution The required cost. This invention uses the squared Euclidean distance as the cost function to accurately reflect the differences between spatial locations:

[0104]

[0105] in, and They are the locations and Two-dimensional spatial coordinates.

[0106] The OT loss is defined as follows:

[0107]

[0108] It is a sufficiently small positive number, the optimal transmission loss Defined as two normalized distributions and Based on the cost matrix Wasserstein distance. To meet the requirements of end-to-end training in deep learning, this invention employs an entropy-regularized optimal transport method and utilizes the Sinkhorn iterative algorithm for efficient and differentiable approximation calculation. This algorithm outputs an approximate value of the OT loss and its relation to the prediction density map through alternating row and column normalization iterations. gradient

[0109] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions.

[0110] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0111] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0112] The above embodiments should be understood as illustrative only and not as limiting the scope of protection of the present invention. After reading the description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A meta-learning and multi-modal large model-based intelligent education scene crowd counting method, characterized in that, The method comprises the following steps: An image to be detected is acquired; the image to be detected is input to a pre-trained double-branch crowd counting model; a first crowd count prediction value is obtained by performing feature extraction and point regression processing on the image through an up-down branch of the double-branch crowd counting model; a second crowd count prediction value is obtained by performing multi-modal feature fusion and density map regression processing on the image through a down-up branch of the double-branch crowd counting model; a third crowd count prediction value is obtained by fusing features of the up-down branch and the down-up branch through a feature fusion module and based on the fused features; and a final crowd counting result is obtained by performing weighted summation on the first crowd count prediction value, the second crowd count prediction value and the third crowd count prediction value through a weighted fusion module.

2. The people counting method of claim 1, wherein, The step of obtaining the first crowd count prediction value through the up-down branch comprises: extracting second layer, third layer and fourth layer features of the input image using a VGG16 backbone network; performing down-sampling on the second layer features, performing up-sampling on the fourth layer features, and performing splicing fusion on the third layer features and the fourth layer features to form up-down branch features; predicting two-dimensional coordinates and confidence scores of head points based on the up-down branch features through a point regression head; and associating the predicted points with the real labeled points based on spatial distances between the predicted points and the real labeled points through a greedy matching algorithm, associating the predicted points with the real labeled points, and obtaining the first crowd count prediction value according to the number of matched points.

3. The people counting method of claim 1, wherein, The step of obtaining the second crowd count prediction value through the down-up branch comprises: calling a large language model based on a preset input template through a spatial prior analysis module to parse the input image into a plurality of spatial regions and generate corresponding regional semantic descriptions; processing the semantic descriptions through a text encoder of a vision-language model to generate regional semantic features; extracting visual features of the image through an image encoder of the vision-language model; inputting the regional semantic features and the image visual features into a regional multi-head attention module for fusion, wherein the regional semantic features are used as a query matrix, the image visual features are used as a key-value matrix, and a position bias generated based on normalized longitudinal coordinates of the image is introduced in attention calculation to align the semantic features with corresponding spatial regions in the image, so that the fused features are obtained; and predicting a density map based on the fused features through a density regression module, and performing spatial integration on the density map to obtain the second crowd count prediction value.

4. The people counting method of claim 1, wherein, The weighted fusion module performs weighted summation using learnable weight parameters which are automatically optimized through gradient descent in a training process of the double-branch crowd counting model.

5. The people counting method of claim 1, wherein, The method also includes a meta-learning training method for a crowd counting model, comprising: obtaining crowd counting datasets from multiple scenes, and constructing a meta-training task set based on the datasets, wherein each task contains a support set and a query set of a scene; performing an outer-loop meta-optimization process: sampling a batch of tasks from the meta-training task set, for each task: performing one or more steps of gradient descent on the current main parameters of the model based on a multi-task loss function on the support set to obtain task-adaptive parameters; calculating the loss on the query set using the task-adaptive parameters; updating the main parameters of the model based on the loss calculated on the query set; and converging the main parameters of the model by iteratively performing the outer-loop meta-optimization process to obtain a meta-initial model with cross-scene rapid adaptation capability.

6. The people counting method of claim 3, wherein, The regionalized semantic features and image visual features are input into a regionalized multi-head attention module for fusion, and the specific steps include: S21 : receiving each image patch feature X extracted by the visual encoder and regionalized text feature extracted by the text encoder ; represents the feature of the text description corresponding to each image patch generated by the LLM in the feature space; S22: converting the regionalized text features T into a Query matrix Q through a linear projection layer; and converting the image region features X into a Key matrix K and a Value matrix V through linear projection layers, respectively; S23: for each image patch, according to its normalized longitudinal coordinate , the bias value corresponding to each attention head h is generated by an independent multi-layer perception MLP network , which together constitutes the position bias matrix ; S24: adding a position bias to an attention scoring function to calculate the interaction weight between each regional text token and all image patches: where each regional text feature vector is used as an independent Query to focus on the corresponding spatial region in the image; is the dimension of each key vector in the key matrix K, which is used as a scaling factor here. is the position bias matrix composed of the position bias of all attention heads and all image patches. is the position bias matrix composed of the position bias of all attention heads and all image patches. S25: The weighted features of the attention output are added to the original image patch features X through a residual connection to obtain the fused cross-modal feature representation ; 。 7. The people counting method of claim 5, wherein, The multi-task loss function includes a point regression loss, a counting loss, and an optimal transport loss.

8. The people counting method of claim 5, wherein, When updating the model parameters on the support set, the MAML algorithm is used, and the optimized parameters include at least one of a learnable semantic prompt vector, a point regression head weight, a density regression head weight, and a weighted fusion weight.

9. An electronic device comprising: one or more processors; a memory; one or more computer programs; wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and the one or more computer programs include instructions for executing the method of any one of claims 1-8. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the crowd counting method of any one of claims 1-8.

Images