Visual Language Modeling with Head Pose Grounding Tasks

Abstract

A multi-stage processing method is presented to generate a visual language model (VLM) for head pose estimation (HPE). [Solution] The method includes the steps of training a first visual language model (VLM) on a first set of images including images of real people to create a head bounding box detection capability; tuning the first VLM using a second set of images including task-specific HPE images to generate a second VLM that is an HPE-oriented VLM; performing layer-based merging of versions of the first VLM and the second VLM by selecting entire layers from either to create a layer-based VLM, wherein the version of the first VLM is the first VLM before training on the first set of images; and fine-tuning the layer-based VLM.

Description

[Technical Field]

[0001] TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to visual language models (VLMs), and more particularly, embodiments disclosed herein relate to integrating head pose estimation (HPE) into VLMs. [Background technology]

[0002]

[0002] HPE is essential in various applications, including driver assistance, human-robot interaction, and customer behavior analysis. Current convolutional neural network (CNN)-based methods, such as 6DRepNet and WHENet, rely on cropped proximity images, where the human head region is first cropped, and often fail in real-world scenarios due to dataset limitations such as a narrow range of head orientation and uniform background. Previous VLM-related works, such as CLIP-Gaze ​​and CLIPose, use VLM embedding for downstream tasks to address pose estimation tasks and rely on external modules or point cloud data, which limits their capabilities.

[0003]

[0003] CogVLM, a state-of-the-art grounded visual language model, demonstrates robust object localization capabilities by generating bounding boxes (BBoxes) in the format [[x0, y0, x1, y1]]. This grounding capability allows CogVLM to focus on specific regions of interest within an image, making it highly effective for tasks requiring spatial precision, such as object detection and referring expression understanding. Summary of the Invention

[0004]

[0004] Methods and apparatus for creating and using a VLM for HPE are disclosed. Some embodiments include a multi-stage processing method for generating a VLM for HPE, the processing method including: training a first VLM on a first set of images to create a head bounding box detection capability, where the first set of images includes images of real people; and tuning the first VLM using a second set of images to generate a second VLM that is an HPE-oriented VLM, where the second set of images differs from the first set of images and includes task-specific HPE images. The processing method also includes layer-based merging of versions of the first and second VLMs by selecting entire layers from either one to create a layer-based VLM, where the first VLM version is the first VLM before training on the first set of images; and fine-tuning the layer-based VLM.

[0005] Other aspects and advantages of the embodiments will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the described embodiments.

[0006] The described embodiments and their advantages can be best understood from the following description read in conjunction with the accompanying drawings, which are in no way intended to limit any changes in form and detail that may be made to the described embodiments by those skilled in the art without departing from the spirit and scope of the described embodiments. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 shows the prompts and responses designed for the HPE task, and also shows some examples of invalid answers. [Figure 2A] 1A-1C illustrate several embodiments of a framework for integrating HPE tasks into the original grounded CogVLM. [Figure 2B] FIG. 1 is a data flow diagram of some embodiments of a multi-stage processing method for generating a visual language model (VLM) for head pose estimation (HPE). [Figure 2C] FIG. 1 is a data flow diagram of some embodiments of a processing method for performing layer-based merging based on a cosine similarity merging criterion. [Figure 3] Figure 1 shows a detailed overview of the different datasets used in our framework. [Figure 4] FIG. 1 shows a comparison of our HPE-CogVLM performance with various baselines. [Figure 5] FIG. 1 illustrates the effect of catastrophic forgetting when data-free rehearsal is applied. [Figure 6] FIG. 1 illustrates the alleviation of the catastrophic forgetting problem under various rehearsal rates. [Figure 7] Figure 1 shows that the HPE-oriented CogVLM model exhibits the highest HPE performance within our framework. [Figure 8A] FIG. 10 is a visualization showing the cross-attention map generated in response to our custom prompts. [Figure 8B] FIG. 10 is a visualization showing the cross-attention map generated in response to our custom prompts. [Figure 9] FIG. 1 depicts an example machine learning architecture used to train a machine learning model. [Figure 10] 1 is a block diagram of some embodiments of a computing device. DETAILED DESCRIPTION OF THE INVENTION

[0008]

[0019] In the following description, numerous details are set forth to provide a more thorough explanation of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present disclosure.

[0009]

[0020] In some embodiments, a robust framework natively enables visual language models (VLMs) to perform head pose estimation (HPE). In some embodiments, the framework integrates HPE into VLMs, such as CogVLM, by leveraging their object detection grounding capabilities. The techniques disclosed herein are not limited to the use of CogVLMs; other VLMs may be used.

[0010]

[0021] Directly adapting a VLM to HPE tasks presents unique challenges. First, integrating HPE functionality into a VLM often leads to invalid outputs. For example, instead of returning a valid format, such as a bounding box at [[x0,y0,x1,y1]], referred to herein as a BBox, or HPE as {yaw, pitch, roll}, the model often produces a mixed format, such as [[x0,y0,yaw angle]]. These mixed outputs are unusable and undermine the effectiveness of the model. Second, integrating HPE functionality into a VLM can result in catastrophic forgetting. More specifically, fine-tuning a VLM for HPE tasks often results in the loss of previously learned object detection and grounding capabilities, limiting the model's versatility for multimodal tasks. Third, existing merging methods for adapting a VLM, such as LoRA fine-tuning or task computation, struggle to balance new task integration with preserving underlying knowledge. These methods either impair the accuracy of the new HPE task or degrade the performance of the original task, such as visual grounding. Fourth, unlike traditional visual grounding tasks, which output 2D bounding boxes, HPE requires precise numerical prediction of 3D Euler angles (yaw, pitch, and roll), introducing challenges in achieving both accuracy and consistency.

[0011]

[0022] Embodiments of the framework disclosed herein enhance the accuracy and robustness of HPE by directly augmenting a VLM (e.g., CogVLM) to natively perform HPE, retaining underlying grounding knowledge, while significantly improving accuracy and robustness. This dual enhancement transforms the VLM into a more advanced and versatile tool capable of simultaneous visual grounding and HPE. This approach bridges the gap between traditional HPE methods and modern multimodal frameworks by integrating HPE capabilities directly into the core of the VLM.

[0012]

[0023] In some embodiments, a layer-based model merging method is utilized to achieve the integration of HPE into the VLM. In some embodiments, merging is performed using cosine similarity with a high cosine similarity threshold and a "winner-take-all" layer selection strategy. By using a high cosine similarity threshold and a "winner-take-all" layer selection strategy, the model's interest is aligned with the HPE task and its basic object detection knowledge is preserved. Furthermore, the use of these techniques solves the problem of mixed and invalid response formats and significantly improves accuracy.

[0013]

[0024] Some embodiments utilize one or more of the following features: prompt design, multi-stage processing methods, layer merging, direct integration over embedding, and mitigation of catastrophic forgetting.

[0014]

[0025] Regarding prompt design, in some embodiments, the prompts leverage full image information (as opposed to cropped images) and BBox coordinates to specify human heads of interest in multi-person scenarios. In some embodiments, the use of such prompts facilitates automated reasoning, reduces the requirement for manual annotation, and improves task robustness through self-attention and cross-attention mechanisms.

[0015]

[0026] Regarding the multi-stage processing method, in some embodiments, the stages include pre-training, fine-tuning, merging, short-term continuous fine-tuning, and evaluation of the model for the HPE task. Each of these stages is discussed in more detail below. In some embodiments, these stages are designed to address specific challenges, such as high HPE accuracy, a valid output format, and preserving the original grounding ability. In some embodiments, the multi-stage processing method uses LoRA-based layer merging with a "winner-takes-all" strategy and a high cosine similarity threshold to match the model to the new task without losing previously acquired knowledge.

[0016]

[0027] In some embodiments, direct integration over embedding is included to enable simultaneous BBox detection and Euler angle prediction for head pose to avoid dependency on any external module.

[0017]

[0028] Finally, in some embodiments, the VLM with integrated HPE mitigates the problem of catastrophic forgetting through the use of a selected rehearsal ratio during continuous fine-tuning to balance retention of old knowledge with learning new grounding tasks.

[0018]

[0029] Each of these features is discussed in more detail below.

[0019] Prompt Design

[0030] In some embodiments, a system using VLM with integrated HPE employs prompts that enable HPE tasks by leveraging information from the full image. In some embodiments, the prompt design utilizes BBox coordinates to designate human heads of interest, especially in scenarios involving multiple individuals. This design automates the inference process and effectively focuses on specific heads, reducing the need for manual annotation. In some embodiments, the system incorporates global features from the self-attention mechanism and head-specific features from the cross-attention mechanism to enhance the robustness of the HPE task.

[0020]

[0031] In some embodiments, the prompts are structured to enable a sophisticated system to respond to specific queries. FIG. 1 shows examples of such representative prompts and responses for the HPE and BBox prediction tasks. Referring to FIG. 1, the prompts include designed prompts such as, for example, "How many human heads are in this image and what is the head bounding box?" as part of the BBox prediction task, and designed prompts such as, for example, "What are the head yaw, pitch, and roll within the bounding box [[106, 168, 148, 242]]?" as part of the head pose estimation task. In some embodiments, the BBox format conforms to the CogVLM specification, and the Euler angle output is formatted as a three-character string as a positive floating-point number, rounded to the nearest integer, and zero-padded when necessary.

[0021] Framework Overview

[0032] Figure 2A illustrates some embodiments of a framework for integrating HPE tasks into a VLM. In some embodiments, a multi-stage integration process for HPE tasks is integrated into the original grounded CogVLM with information from dataset usage, designed prompts, and model merging strategies to generate a model referred to herein as the HPE-CogVLM. Each stage of this framework enhances different aspects of the model's capabilities and progressively refines parameters to balance HPE and BBox tasks.

[0022]

[0033] In some embodiments, the fine-tuning process at each stage continues with the fine-tuning of the CogVLM, which implements LoRA, a well-known algorithm in the art, across transformation blocks, including query, key, attention, and densely connected layer values. The LoRA matrices for each layer are then accumulated in the corresponding layer within the original grounded CogVLM. To support this multi-stage processing method, the framework utilizes a variety of datasets, each of which plays a unique role in model training and evaluation. Figure 3 provides an overview of how these datasets, used in some embodiments, contribute across multiple stages to efficiently train and evaluate the model. In some embodiments, the use of such datasets can enhance head BBox detection, improve HPE task accuracy, and prevent catastrophic forgetting.

[0023]

[0034] Referring to Figure 2A, the process refines parameters through pre-training, supervised fine-tuning, layer-based merging, continuous fine-tuning, and final evaluation stages. The process strategically utilizes a specific dataset, shown in Figure 3, across multiple stages to effectively train and evaluate the model.

[0024] Stage 1: Pre-training with weakly labeled data

[0035] In pre-training stage 1, a VLM model, the original grounding CogVLM 202, is trained with weakly labeled images 201 to develop capabilities for human head BBox detection. In some embodiments, the weakly labeled images 201 are from the Crowd Human dataset. Furthermore, this stage serves as a warm-up for the model's HPE capabilities by utilizing weakly labeled images. In other words, this pre-training prepares the model to begin HPE tasks in real-world scenarios.

[0025]

[0036] To achieve this goal, in some embodiments, the original grounded CogVLM202 is pre-trained on a crowd-human dataset using pseudo-labels derived from a pre-trained 6DRepNet model consisting of images of actual people in various scenes to enable real-world human head detection and to initiate the HPE task. The crowd-human dataset provides a rich collection of human head images in various poses and situations. This approach enables CogVLM202 to accurately locate the BBox of a human head and establishes a baseline understanding for the subsequent HPE task. However, since the original crowd-human dataset only provides accurate head BBox annotations and lacks ground truth (GT) annotations for HPE, weak HPE annotations can be inferred using 6DRepNet. The output model from this stage is called the weakly labeled CogVLM204, as shown in Figure 2A.

[0026] Phase 2: Supervised fine-tuning on task-specific HPE data

[0037] In Phase 2, the weakly labeled CogVLM 204 undergoes fine-tuning. Following the pre-training phase 201, the weakly labeled CogVLM 204 proceeds to a supervised fine-tuning phase, in which the weakly labeled CogVLM 204 is trained exclusively on task-specific HPE images 203. As shown in FIG. 3, in some embodiments, fine-tuning in Phase 2 is performed using a synthetic dataset of images. In some embodiments, the synthetic dataset is the Agora dataset as the task-specific images. Unlike the broader Crowd Human dataset used in Phase 1, the Agora dataset consists of synthetic images adapted to capture precise head pose information, resulting in more accurate annotations. More specifically, the synthetic Agora dataset provides precise annotations, including ground truth (GT) for SMPLX parameters and full-range east yaw angles, for the fine-tuning HPE task. The detailed annotations in the Agora dataset enable the weakly labeled CogVLM 204 to focus on subtle variations in head orientation, enhancing its ability to make fine-grained predictions.

[0027]

[0038] In some embodiments, Stage 2 focuses on improving HPE accuracy, addressing weaknesses in the model's performance due to low-quality annotations. In some embodiments, fine-tuning with a synthetic dataset remedies the limitations of weakly labeled annotations. The output model is called the HPE-oriented CogVLM206, as shown in FIG. 2A. This refined HPE-oriented CogVLM206 combines real-image exposures and contextual background knowledge obtained from Stage 1 with task-specific accuracy obtained in Stage 2.

[0028] Stage 3: Layer-based merging

[0039] During stage 3, layer-based merging occurs. In some embodiments, during layer-based merging, the original CogVLM 202 is merged with the HPE-oriented CogVLM 206 (from stage 2). In some embodiments, the layers of both VLMs are received by a merge criteria processing block 207, which selects layers from either or both the original CogVLM 202 and the HPE-oriented CogVLM 206 for inclusion in the layer-based merged CogVLM 220 based on the merge criteria.

[0029]

[0040] In some embodiments, layer-based merging is performed based on a cosine similarity criterion, where cosine similarity is used to measure the amount of information shared between layers. In some embodiments, cosine similarity is calculated as the average cosine similarity between the layer parameter tensors of the original grounding CogVLM 202 and the HPE-oriented CogVLM 206 along the last dimension.

[0030]

[0041] In some embodiments, a cosine similarity threshold is used to determine whether a layer from the HPE-oriented CogVLM 206 should be merged into the final, layer-based model (e.g., the layer-based merging CogVLM 220). In some embodiments, because LoRA fine-tuning is applied in a previous stage, most of the original model parameters are only minimally changed, necessitating a higher threshold for cosine similarity. In some embodiments, the threshold is set to 0.95. This can ensure that layers with substantial information overlap are merged. In some other embodiments, the threshold is a value greater than 0.9. If the similarity falls below this threshold, the original knowledge is fully retained (e.g., a layer from the original grounding CogVLM 202 is selected for inclusion in the resulting layer-based model after merging (e.g., the layer-based merging CogVLM 220)). Otherwise, if the similarity exceeds a threshold, this indicates substantial overlap of information by strict criteria, and an entire layer from the HPE-oriented CogVLM 206 is selected for inclusion in the layer-based model to ensure minimal risk of losing important existing knowledge.

[0031]

[0042] For example, layer 210 is from the original grounding CogVLM 202, and layer 211 is from the HPE-oriented CogVLM 206. As shown, layer 1 from layer 211 of the HPE-oriented CogVLM 206 is merged into the layer-based merging CogVLM 220, along with layers 2, 3, and n from layer 210 of the original grounding CogVLM 202. Note that in some embodiments, a layer in such a model includes at least one matrix, as is well known in the art.

[0032]

[0043] In some embodiments, the merging criteria for use in some embodiments are detailed as follows: 1) Calculate and rank the cosine similarities across all layers from both models, always selecting layers from the original grounding CogVLM 202 that are within the smallest 1% of cosine similarity, and the ranking is used to identify the smallest 1% of cosine similarities. In some other embodiments, it is 5% or less. 2) Select a layer from the original grounding CogVLM 202 for inclusion in the layer-based model when the cosine similarity between two layers from each model is below a threshold (e.g., layer-based merging CogVLM 220). 3) If not, select a layer from the HPE-oriented CogVLM 206 for inclusion in the layer-based model (e.g., Layer-Based Merging CogVLM 220).

[0033]

[0044] Therefore, a high cosine similarity threshold is used, and a "winner-take-all" approach is used to select entire layers from either the original grounding CogVLM 202 or the layer-based model, the HPE-oriented CogVLM 206, to create the merging CogVLM. This allows the attention of the merged model to be aligned to the new HPE task at reduced and potentially minimal cost. Furthermore, the "winner-take-all" strategy prevents mixing errors in the output structure and enhances task-specific focusing while maintaining robustness. Furthermore, by applying a high similarity threshold, layers with minimal similarity are retained from the original grounding CogVLM 202 to preserve foundational knowledge, and new knowledge is introduced only when there is substantial information overlap between the two models, ensuring that existing knowledge is well preserved even when layers are selected from the HPE-oriented CogVLM 206. This approach reduces attention loss while aligning the model to the new task. This new capability introduced in Stage 3, while seemingly basic, provides a strong foundation for further refinement, allowing the model to effectively build on this baseline in Stage 4. Furthermore, by incorporating entire layers rather than individual parameters, the method preserves the structural integrity of the model and reduces the risk of response mixing when dealing with multiple grounding tasks with different requirements.

[0034]

[0045] 2A also shows designed prompts 205. In some embodiments, the designed prompts 205 include, for example, "How many human heads are in this image and what is the head bounding box?" and "What is the head yaw, pitch, and roll within the bounding box [[111,222,333,444]]?"

[0035] Stage 4: Continuous fine-tuning for mixed data

[0046] After merging, the layer-based merging CogVLM 220 undergoes an additional period of fine-tuning with images 214, including both task-specific HPE images and rehearsal images. In some embodiments, the optimal rehearsal ratio for training rehearsal images is predefined during Stage 1 by conducting several parallel experiments. In these experiments, the original grounding CogVLM 202 is trained with weakly labeled images, each combined with various percentages of rehearsal images (e.g., 0%, 1%, 10%, 25%), and the rehearsal ratio that produces the best performance is then used to fine-tune the merged model in this stage. In some embodiments, the percentage of rehearsal images used to fine-tune the layer-based merging CogVLM 220 is 10%. However, the technique is not limited to using 10%. For example, in some embodiments, other percentages may be used, including, but not limited to, a range extending several percentage points on either side of 10%.

[0036]

[0047] Unlike the fine-tuning in Stage 2, this phase involves only a short period of fine-tuning, less than one epoch. The rationale for incorporating an additional short fine-tuning is that layer merging preserves parameter integrity and optimizes model focus, but lacks the fine-tuned parameters necessary to enhance HPE prediction accuracy. The optimal rehearsal rate, determined during pre-training, reduces and potentially minimizes catastrophic forgetting and improves the model's numerical accuracy. In this way, the merging model can be rapidly fine-tuned to yield accurate numerical predictions. The final output model of this stage is HPE-CogVLM221, as shown in Figure 2A.

[0037]

[0048] In some embodiments, the rehearsal datasets used for fine-tuning include the Refcoco, Refcoco+, and Refcocog datasets used for BBox prediction tasks. The use of these datasets helps mitigate catastrophic forgetting through controlled rehearsal rates determined by empirical evaluation.

[0038] Stage 5: Evaluation

[0049] In Stage 5, the final model, HPE-CogVLM 221, is evaluated against real-world images 215 for the HPE task and against a rehearsal dataset for BBox prediction. In some embodiments, the real-world images are a subset of the CMU panoptic images. The subset of the CMU panoptic dataset evaluates the model's HPE task performance and reflects real-world conditions. The rehearsal dataset is used to validate BBox localization capabilities. In some other embodiments, other sets of real-world or non-real-world images may be used.

[0039]

[0050] In some embodiments, four evaluation metrics are used to assess the HPE and BBox prediction tasks. These metrics are: angular error ratio (E angle ), BBox error ratio (E bbox ), Precision (Acc.), and Mean Absolute Error (MAE). Note that in other embodiments, a subset of these metrics are used to evaluate HPE-CogVLM221. In still other embodiments, additional metrics beyond the four metrics are used to evaluate HPE-CogVLM221.

[0040] 1) Angular error ratio (E angle ) - Evaluate invalid HPE format predictions. In some embodiments, E angle =e angle / t angle and e angle indicates an invalid HPE response, and t anglemeans the total number of HPE responses. This metric is defined to assess the model's ability to produce appropriate numerical outputs for HPE tasks. When prompted with HPE queries, CogVLM may generate inappropriate responses, such as natural language processing (NLP) task responses like "human head," BBox task responses like "[[111,222,333,444]]," or mixed responses like "[[111,999,999,99]]," as shown in Figure 1.

[0041] 2) BBox error ratio (E bbox ) - Evaluate invalid BBox format predictions. In some embodiments, E bbox =e bbox / t bbox and e bbox means the number of invalid BBox answers, and t bbox means the total number of BBox answers. This metric assesses the model's ability to provide a good numerical output for the BBox prediction task.

[0042] 3) BBox Accuracy (ACC.) - Measures correct BBox detection with IoU>0.5. In some embodiments, Acc.=m / m^, where m means the number of valid BBox answers with IoU>0.5 and m^ means the total number of valid BBox answers. A BBox prediction is considered accurate if the intersection over union (IoU) between the GT and the prediction is greater than 0.5. Invalid answers are excluded from the accuracy and MAE calculations.

[0043] 4) MAE of Euler angles (MAE) - measures the accuracy of Euler angles. For the HPE task, the MAE between the GT Euler angles and the predicted Euler angles is defined as:

number

number

[0044]

[0051] 2B shows a data flow diagram of some embodiments of a multi-stage processing method for generating a visual language model (VLM) for head pose estimation (HPE). The processing method may be performed by processing logic, which may include hardware (e.g., circuitry, dedicated logic, memory, etc.), software (such as running on a general-purpose computer system or dedicated machine), firmware (e.g., software programmed in read-only memory), or a combination thereof.

[0045]

[0052] 2B , the processing method includes processing logic training a first visual language model (VLM) on a first set of images, the first set of images including images of real people, to develop a head bounding box detection capability (processing block 231). In some embodiments, the training is considered pre-training of the VLM. In some embodiments, the first VLM is a CogVLM, the first set of images including weakly labeled images, and the second set of images including synthetic images.

[0046]

[0053] After training the first VLM with the first set of images, processing logic tunes the first VLM with a second set of images to generate a second VLM that is an HPE-oriented VLM (processing block 232). In some embodiments, the second set of images differs from the first set of images and includes task-specific HPE images.

[0047]

[0054] After generating the HPE-oriented VLM, processing logic performs layer-based merging by selecting entire layers from either a version of the first VLM or the second VLM to create a layer-based VLM (processing block 233). For this operation, in some embodiments, processing logic uses a previously existing version of the first VLM that was pre-trained on the first set of images. In some embodiments, performing layer-based merging of the first VLM and the second VLM includes determining the amount of information shared between layers of the first VLM and the second VLM and merging layers with information overlap exceeding a first threshold.

[0048]

[0055] In some embodiments, the layer-based merging is based on a cosine similarity merging criterion. In some embodiments, the processing method further includes calculating a cosine similarity across all layers of the first and second VLMs as the average cosine similarity between the layer parameter tensors of the first and second VLMs. In some embodiments, the processing method further includes calculating the cosine similarity across all layers of the first and second VLMs, ranking the cosine similarities (e.g., layers within the smallest 1% of cosine similarity) for use in selecting layers from the first VLM, and selecting one layer from each pair of corresponding layers of the first and second VLM layers to be part of the layer-based VLM based on a comparison between the cosine similarity and a first threshold. This processing method is shown in FIG. 2C. In some embodiments, the step of choosing one layer from each pair of corresponding layers of the first and second VLM layers includes selecting a layer from the first VLM upon determining that the cosine similarity between the two corresponding layers of the first and second VLM is less than a threshold, and selecting a layer from the second VLM upon determining that the cosine similarity between the two layers from each model is greater than a threshold.

[0049]

[0056] After creating the layer-based merging VLM, processing logic fine-tunes the layer-based VLM (processing block 234). In some embodiments, fine-tuning the layer-based VLM is via task-specific HPE images from the second set of images and the set of rehearsal images for BBox prediction.

[0050]

[0057] After fine-tuning the layer-based VLM, processing logic evaluates the layer-based VLM on the test data for the HPE task and on the rehearsal data set for BBox prediction (processing block 235).

[0051]

[0058] In some embodiments, the processing method also includes querying the layer-based VLM with an HPE prompt (processing block 236). In some embodiments, the query occurs after fine-tuning the layer-based VLM. In some embodiments, the HPE prompt includes full image information and bounding box coordinates to specify a human head of interest in a multi-person image.

[0052]

[0059] Experimental results, as evidenced in Figure 4, demonstrate that the HPE-CogVLM created with the multi-stage processing method demonstrates significant improvements over traditional CNN-based models and alternative VLM-based baselines for HPE tasks and BBox prediction. HPE-CogVLM exhibits significantly lower MAE compared to traditional CNN-based models, such as WHENet, HopeNet, and 6DRepNet. Specifically, MAE is reduced by 75.1%, 66.8%, and 31.5%, respectively. These results clearly demonstrate the superior robustness and accuracy of HPE-CogVLM compared to CNN-based approaches, which are inferior to other VLM-based models. This demonstrates the inherent advantages of VLM for tasks requiring robustness and multimodal grounding. Compared to non-merged CogVLM, HPE-CogVLM achieves a 10% lower MAE compared to non-merged CogVLM. Furthermore, our E angleis 2.5 times smaller, indicating its enhanced proficiency in the HPE task. This demonstrates the effectiveness of the LoRA layer-based merging method used by the present invention, which significantly improves task-specific performance compared to approaches that do not utilize any model merging techniques. The BBox prediction accuracy of HPE-CogVLM on the test dataset is slightly lower than that of the non-merged CogVLM by 0.6%, 0.5%, and 1.1%, respectively, and this result is achieved with only one-fifth the rehearsal image training iterations required by the non-merged CogVLM. This highlights the efficiency of HPE-CogVLM in achieving comparable BBox accuracy while using significantly fewer computational resources, further reinforcing its practical advantages. Compared to the task-acceleration (TA) merging CogVLM, HPE-CogVLM outperforms the TA merging CogVLM across all evaluated metrics. In particular, the BBox prediction accuracy of HPE-CogVLM exceeds that of TA-merging CogVLM by 1%, 2.4%, and 1.7% on the test dataset. angle is 68.9%, 1.325 times larger than that of the present invention, indicating that only 31.1% of the TA-merged CogVLM's responses to the HPE task are valid, invalidating the MAE metric for evaluation. These results demonstrate the inability of the TA-merged CogVLM to generate appropriate numerical responses, even after additional fine-tuning.

[0053]

[0060] Furthermore, regarding catastrophic forgetting in HPE tasks, the phenomenon of catastrophic forgetting is evident in models trained exclusively on HPE tasks using the Agora dataset. Figure 5 shows a continuous decline in performance on previously acquired tasks, such as object detection, during adaptation to a new HPE task. The results demonstrate a distinct pattern of catastrophic forgetting, in which previously acquired knowledge in BBox prediction is substantially diminished before the model solidifies its understanding of the new HPE task. This behavior contrasts with human cognitive processes, in which new and old knowledge often coexist and can complement each other. In human learning, integrating new information with existing knowledge is typically achieved without the catastrophic forgetting observed in machine learning models.

[0054]

[0061] However, in some embodiments, the rehearsal rate is used to mitigate catastrophic forgetting. In some embodiments, a systematic approach is used to determine the optimal rehearsal rate to mitigate catastrophic forgetting on the BBox prediction task during fine-tuning. Figure 6 shows a detailed comparison of the performance of the weakly labeled CogVLM at various rehearsal rates (0%, 1%, 10%, and 25%) during Phase 1 of training. This analysis identifies the most effective rate for retaining original task knowledge during the transition to a new task. In some embodiments, a 10% or 25% rehearsal rate is selected for the Phase 4 fine-tuning experiment. These rates are significantly higher than the 1% rehearsal rate commonly used in non-grounding tasks, highlighting the unique requirements of grounding tasks, such as BBox prediction and HPE. There is a trade-off between retaining prior knowledge and acquiring new task competence. Higher rehearsal rates benefit preserving existing skills, but at the expense of performance on the new task. Conversely, a lower rehearsal rate may enhance new task performance but slightly reduce retention of prior knowledge. After evaluating both factors, in some embodiments, a 10% rehearsal rate is the optimal balance. This rate achieves significantly better HPE performance with only a negligible decrease in BBox prediction accuracy compared to higher rates. The HPE-CogVLM model trained with a 10% rehearsal rate is thus selected as the optimal model, effectively balancing the need to retain existing knowledge while enhancing new task performance.

[0055]

[0062] Figure 7 shows the performance of the HPE-oriented CogVLM on the HPE task. The HPE-oriented CogVLM, created in Phase 2 of our framework, is specifically designed for the HPE task and provides optimal performance without addressing BBox prediction capabilities. The comparison results shown in Figures 8A and 8B highlight the superior performance of the HPE-oriented CogVLM compared to traditional CNN-based models such as 6DRepNet.

[0056]

[0063] The technology disclosed herein includes a novel approach to visualizing cross-attention maps in response to specially designed prompts, demonstrating precise localization capabilities within images containing multiple individuals. Figures 8A and 8B provide visual evidence of the model's ability to focus attention based on BBox inputs within the prompt. Referring to Figure 8A, the attention map associated with the prompt "What is the head yaw, pitch, and roll within the bounding box (BBox) [[335, 179, 445, 332]]?" is shown, visualizing the model's response to the prompt for the HPE task with the specified BBox [[335, 179, 445, 332]]. The cross-attention map highlights the head of the individual on the left, demonstrating precise localization and task-specific focus. Referring to Figure 8B, we similarly show the attention map associated with the prompt "What is the head yaw, pitch, and roll within the bounding box (BBox) [[775,105,893,261]]?" and the model's response to the prompt specifying a BBox [[775,105,893,261]] so that the model accurately targets an individual's head. These visualizations validate the model's spatial awareness by confirming its ability to localize attention to specified regions within an image. This ability ensures precise task execution, such as HPE or BBox prediction, even in complex multi-person scenarios.

[0057]

[0064] Furthermore, the techniques disclosed herein demonstrate that CogVLM can effectively process and respond to BBox inputs specified within prompts: it can accurately interpret pre-specified BBoxes provided within prompts, ensuring robust multimodal grounding for visual-linguistic tasks.

[0058]

[0065] Therefore, the technology disclosed herein offers several novel features. First, the technology includes a method for integrating HPE tasks into VLMs through a multi-stage processing method applicable to VLM architectures, including pre-training, supervised fine-tuning, layer-based merging, and short-term continuous fine-tuning. Second, the technology includes a model merging technique using a cosine similarity threshold combined with a winner-take-all strategy, enabling the integration of task-specific knowledge while preserving underlying capabilities and applicable to the merging of more than two models. Third, the technology includes direct integration of HPE into VLMs, going beyond embedded extraction methods and enabling simultaneous object detection and precise head pose estimation. Fourth, the technology includes mitigation of catastrophic forgetting through an optimized rehearsal rate (10%) during continuous fine-tuning for grounding tasks. Fifth, the technique includes a prompt design method that leverages full image information and BBox coordinates to specify human heads of interest in multi-person scenarios, facilitating automated inference, reducing the requirement for manual annotation, and improving task robustness through self-attention and cross-attention mechanisms.

[0059]

[0066] The HPE-CogVLM framework and the techniques disclosed herein can be applied to several applications, including, but not limited to, driver monitoring systems for real-time attention estimation, surveillance systems for crowd behavior analysis, augmented reality and virtual reality systems requiring precise head tracking, advanced robotics for human-robot interaction, and retail environments for customer gaze tracking and behavior analysis. Its modular architecture allows for seamless integration with existing systems, enhancing versatility and scalability.

[0060] Example machine learning model

[0067] 9 illustrates an example machine learning architecture 900 used to train a machine learning model 902. An input module 904 receives input

number

number

number

number

number

number

number

number

number

number

[0061]

[0068] In some embodiments, the input

number

number

number

number

number

number

number

number

[0062]

[0069] In some machine learning models, all layers of the model are fully connected. For example, in an MLP model, all perceptrons are

number

[0063] Example Device

[0070] FIG. 10 shows a block diagram of several embodiments of a computing device 1000 capable of performing one or more of the operations described herein. The computing device 1000 may be connected to other computing devices within a local area network (LAN), an intranet, an extranet, and / or the Internet. The computing device may operate in the capacity of a server machine in a client-server network environment or in the capacity of a client in a peer-to-peer network environment. The computing device may be provided by a personal computer (PC), a server computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, an ultrasound machine, or any machine capable of executing (sequentially or otherwise) a set of instructions that define actions to be taken by the machine. Furthermore, although only a single computing device is shown, the term “computing device” shall be interpreted to include any collection of computing devices that individually or together execute a set (or sets) of instructions to perform the methods discussed herein.

[0064]

[0071] The example computing device 1000 may include a processing device 1002 (e.g., a general-purpose processor, a programmable logic device (PLD), etc.), a main memory 1004 (e.g., a synchronous dynamic random access memory (DRAM), a read-only memory (ROM), etc.), and a static memory 1006 (e.g., a flash memory, a data storage device 1008, etc.), which may communicate with each other via a bus 1010. The processing device 1002 may be implemented by one or more general-purpose processing devices, such as a microprocessor, a central processing unit, etc. In some embodiments, the processing device 1002 comprises a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or a combination of instruction sets. The processing device 1002 may also comprise one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. The processing device 1002 may be configured to perform the operations described herein in accordance with one or more aspects of the present disclosure to perform the operations and steps discussed herein.

[0065]

[0072] Computing device 1000 may further include a network interface device 1012 capable of communicating with a network 1014. Computing device 1000 may also include a video display unit 1016 (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED), a cathode ray tube (CRT), etc.), an alphanumeric input device 1018 (e.g., a keyboard), a cursor control device 1020 (e.g., a mouse), and an acoustic signal generation device 1022 (e.g., a speaker, a microphone, etc.). In one embodiment, the video display unit 1016, the alphanumeric input device 1018, and the cursor control device 1020 may be combined into a single component or device (e.g., an LCD touchscreen).

[0066]

[0073] The data storage device 1008 may include a computer-readable storage medium 1024 on which one or more sets of instructions 1026 (e.g., instructions for performing operations described herein in accordance with one or more aspects of the present disclosure) may be stored. The instructions 1026 may also reside, completely or at least partially, within the main memory 1004 and / or within the processing device 1002 during their execution by the computing device 1000, with the main memory 1004 and the processing device 1002 also constituting computer-readable media. The instructions may also be transmitted or received over the network 1014 through the network interface device 1012.

[0067]

[0074] Various technologies are described in the general context of software, hardware elements, or program modules. Generally, such modules include routines, programs, objects, elements, components, data structures, etc. that perform particular tasks or implement particular abstract data types. As used herein, the terms “module,” “functionality,” and “component” generally refer to software, firmware, hardware, or combinations thereof. In some aspects, modules described herein are embodied as executable instructions or code within the data storage device 1008 of the computing device 1000. While represented as a software implementation, the described modules may be implemented as any form of control application, software application, signal processing and control module, hardware, or firmware implemented on the computing device 1000.

[0068]

[0075] While the computer-readable storage medium 1024 is shown in the illustrative example as being a single medium, the term "computer-readable storage medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions. The term "computer-readable storage medium" should also be understood to include any medium that can store, encode, or carry a set of instructions for execution by a machine and that cause the machine to perform the methods described herein. The term "computer-readable storage medium" should accordingly be understood to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0069]

[0076] Some portions of the above detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0070]

[0077] It should be borne in mind, however, that all these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specified, or as will become apparent from the discussion that follows, terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" refer to methods of operation and processing of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities in the computer system's registers and memory into other data similarly represented as physical quantities in the computer system's memory or registers or other such information storage, transmission, or display device.

[0071]

[0078] This disclosure also relates to apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a computer-readable storage medium such as, but not limited to, floppy disks, optical disks, CD-ROMs, and magneto-optical disks, read-only memory (ROM), random-access memory (RAM), EPROM, EEPROM, magnetic or optical cards, or any type of medium suitable for storing electronic instructions and each coupled to a computer system bus.

[0072]

[0079] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the description below. Additionally, the present disclosure is not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​may be used to implement the teachings of the present disclosure, as described herein.

[0073]

[0080] A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, machine-readable media include read-only memory ("ROM"), random-access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory devices, electrical, optical, acoustical, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), etc.

[0074]

[0081] While many variations and modifications of the present disclosure will no doubt become apparent to those skilled in the art after reading the foregoing description, it should be understood that it is in no way intended that any particular embodiments shown and described by way of illustration be considered limiting. Accordingly, references to details of various embodiments are not intended to limit the scope of the claims, which themselves recite only those features regarded as essential to the present disclosure.

Claims

1. A multi-stage processing method for generating a visual language model (VLM) for head pose estimation (HPE), comprising: one or more processors training a first visual language model (VLM) on a first set of images to develop a head bounding box detection capability, the first set of images including images of real people; the one or more processors training the first VLM after training using a second set of images to generate a second VLM, the second set of images being different from the first set of images and including task-specific HPE images; performing layer-based merging of the first VLM version and the second VLM version by the one or more processors selecting one or more entire layers from each of the first VLM version and the second VLM version for merging to produce a layer-based VLM, wherein the first VLM version is the first VLM before training on the first set of images; the one or more processors fine-tuning the layer-based VLM; A multi-step processing method comprising:

2. The step of the one or more processors querying a layer-based VLM via an HPE prompt; The multi-step processing method of claim 1 further comprising:

3. The multi-stage processing method of claim 2 , wherein the HPE prompt comprises full image information and bounding box coordinates for specifying a human head of interest in a multi-person image.

4. The multi-stage processing method of claim 1 , wherein the first VLM is a CogVLM, the first set of images includes weakly labeled images, and the second set of images includes synthetic images.

5. The step of performing layer-based merging includes: determining an amount of information shared between the first and second VLM layers; Merging layers having information overlap exceeding a first threshold; 2. The multi-step processing method of claim 1, comprising:

6. The multi-stage processing method of claim 1 , wherein the layer-based merging is based on a cosine similarity merging criterion.

7. The one or more processors calculate a cosine similarity across all layers of the first and second VLMs as an average cosine similarity between layer parameter tensors of the first and second VLMs; The multi-step processing method of claim 6 further comprising:

8. The method of claim 7, wherein the one or more processors calculate a cosine similarity across all layers of the first and second virtual linear models; the one or more processors ranking cosine similarities for use in selecting the layer from the first VLM within a minimum of 1% of cosine similarities; the one or more processors selecting one layer from each pair of corresponding layers of the first and second VLMs to be part of the layer-based VLM based on a comparison between cosine similarity and a first threshold; The multi-step processing method of claim 6 further comprising:

9. The step of selecting one layer from each pair comprises: selecting a layer from the first VLM upon determining that the cosine similarity between two corresponding layers of the first and second VLMs is less than the first threshold; selecting the layer from the second VLM upon determining that the cosine similarity between two layers from each model is greater than the first threshold; 9. The multi-step processing method of claim 8, comprising:

10. 2. The multi-stage processing method of claim 1, wherein the step of fine-tuning the layer-based VLM is performed via task-specific HPE images from the second set of images and a set of rehearsal images for bounding box prediction.

11. After the step of fine-tuning the layer-based VLM, the one or more processors evaluating the layer-based VLM against test data for HPE tasks and against a rehearsal dataset for bounding box prediction; The multi-step processing method of claim 1 further comprising:

12. 1. A system for simultaneous object detection and head pose estimation, comprising: an interface for receiving a head pose estimation (HPE) prompted query; one or more processors coupled to the interface and operable to execute an application for performing head pose estimation for the query using a layer-based visual language model (VLM) for head pose estimation (HPE), the VLM having been generated using a multi-stage processing method; Equipped with The multi-stage processing method includes: training a first visual language model (VLM) on a first set of images to develop a head bounding box detection capability, wherein the first set of images includes images of real people, and the first set of images includes weakly labeled images; training the first VLM after training using a second set of images to generate a second VLM that is an HPE-directed VLM, the second set of images being different from the first set of images and including task-specific HPE images, and the second set of images including synthetic images; performing layer-based merging of the first VLM version and the second VLM version by selecting one or more entire layers from each of the first VLM version and the second VLM version for merging to produce a layer-based VLM, where the first VLM version is the first VLM in its state before training on the first set of images; performing fine tuning of the layer-based VLM; Including, the system.

13. The system of claim 12 , wherein the first VLM is a CogVLM.

14. The system of claim 12 , wherein the HPE prompt comprises full image information and bounding box coordinates for specifying a human head of interest in a multi-person image.

15. 13. The system of claim 12, wherein the step of performing layer-based merging includes determining an amount of information shared between layers of the first and second VLMs and merging layers having information overlap that exceeds a first threshold.

16. The layer-based merging is based on a cosine similarity merging criterion, The step of performing layer-based merging includes: calculating a cosine similarity across all layers of the first and second VLMs as the average cosine similarity between layer parameter tensors of the first and second VLMs; The system of claim 12 further comprising:

17. The layer-based merging is based on a cosine similarity merging criterion, The step of performing layer-based merging includes: calculating a cosine similarity across all layers of the first and second VLMs; ranking the cosine similarities for use in selecting the layers from the first VLM within 1% of the lowest cosine similarities; selecting one layer from each pair of corresponding layers of the first and second VLMs to be part of the layer-based VLM based on a comparison between the cosine similarity and a first threshold; The system of claim 12 further comprising:

18. The step of selecting one layer from each pair comprises: selecting a layer from the first VLM upon determining that the cosine similarity between two corresponding layers of the first and second VLMs is less than the first threshold; selecting the layer from the second VLM upon determining that the cosine similarity between two layers from each model is greater than the first threshold; 20. The system of claim 17, comprising:

19. 13. The system of claim 12, wherein the step of fine-tuning the layer-based VLM is performed via task-specific HPE images from the second set of images and a set of rehearsal images for bounding box prediction.

20. 1. A computer-readable storage medium having executable instructions stored thereon for causing one or more processing devices to perform head pose estimation (HPE) using a layer-based VLM created using a multi-stage processing method, the method comprising: The multi-stage processing method includes: training a first visual language model (VLM) on a first set of images to develop a head bounding box detection capability, the first set of images including images of real people; training the first VLM after training with a second set of images to generate a second VLM that is an HPE-directed VLM, the second set of images being different from the first set of images and including task-specific HPE images; performing a layer-based merging of the first VLM version and the second VLM version by selecting one or more entire layers from each of the first VLM version and the second VLM version for merging to produce a layer-based VLM; performing fine tuning of the layer-based VLM; 1. A computer-readable storage medium comprising:

Images