Automobile door store full-scene quality inspection method based on multi-granularity feature adaptive fusion

Abstract

The application relates to a full-scene quality inspection method for a car store based on adaptive fusion of multi-granularity features, which comprises the following steps: acquiring a video stream and an audio stream in a car store; extracting multi-scale visual features from the video stream, and parallelly intercepting multiple feature branches on the last layer feature map of the network to respectively capture fine-grained interaction, macroscopic body posture trajectory and environmental state semantics; extracting an audio feature vector from the audio stream, and combining the audio feature vector with the visual feature vector to form a multi-granularity feature unit pool; calculating the instance confidence of each feature vector, constructing a gating network, and under the guidance of a scene prompt word, a task prompt word and the instance confidence, allocating feature unit quotas for each feature vector, and then screening a corresponding number of feature units from the pool to be fused into a new feature vector and output; and generating a full-scene quality inspection result according to the feature vector generated by the gating network. The application can efficiently, accurately and interpretively realize full-scene quality inspection of a car store.

Description

Technical Field

[0001] This invention relates to the fields of automotive store quality inspection, artificial intelligence and computer vision technology, and in particular to a full-scenario quality inspection method for automotive stores based on multi-granularity feature adaptive fusion. Background Technology

[0002] Intelligent quality inspection systems are core tools for ensuring service standardization and user experience in the automotive industry. However, existing quality inspection systems suffer from three major problems: fragmentation, rigidity, and being "black boxes."

[0003] 1. Traditional quality inspection systems adopt a "scenario-specific" architecture, with independent quality inspection solutions developed for different scenarios. Multiple models need to be maintained simultaneously. This siloed architecture not only results in high development and maintenance costs but also makes it difficult to expand to new scenarios, failing to meet the needs of full-scenario coverage in car dealerships.

[0004] 2. Existing multimodal fusion solutions mostly adopt static feature fusion strategies, but do not consider the differentiated requirements of features in different scenarios. This leads to redundant interference of environmental features in sales scenarios and inefficient use of micro-interaction features in maintenance scenarios, ultimately causing a decrease in quality inspection accuracy in specific scenarios.

[0005] 3. Existing quality inspection solutions based on large models often only output conclusions in their decision-making process, failing to pinpoint which specific erroneous statement or action caused the violation. This lack of interpretability makes the quality inspection results unsuitable for precise training, requiring trainers to sift through a large number of records to identify problems, resulting in low efficiency. Furthermore, the lack of traceability when the model makes a misjudgment affects the system's credibility. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a full-scenario quality inspection method for automobile stores based on multi-granularity feature adaptive fusion, which can realize full-scenario quality inspection of automobile stores in an efficient, accurate and interpretable manner.

[0007] The technical solution adopted by this invention to solve its technical problem is: to provide a full-scenario quality inspection method for automobile stores based on multi-granularity feature adaptive fusion, including:

[0008] Acquire video and audio streams collected within the car dealership;

[0009] A first feature extraction network is constructed to extract multi-scale visual features from the acquired video stream, and multiple feature branches are extracted in parallel on its last layer feature map, including a first feature branch for capturing fine-grained interactions, a second feature branch for capturing macroscopic body trajectories, and a third feature branch for capturing environmental state semantics.

[0010] A second feature extraction network is constructed to extract audio feature vectors from the acquired audio stream;

[0011] The first visual feature vector generated by the first feature branch, the second visual feature vector generated by the second feature branch, the third visual feature vector generated by the third feature branch, and the feature units contained in the audio feature vector are collected to form a multi-granularity feature unit pool.

[0012] The instance confidence of the first visual feature vector, the second visual feature vector, the third visual feature vector, and the audio feature vector are calculated respectively to quantify the real-time reliability of each feature vector in the actual scene and form a confidence vector.

[0013] Determine scene prompts and task prompts, construct a gating network to allocate feature unit quotas to each feature vector based on scene prompts, task prompts and confidence vectors, and then select a number of feature units from the multi-granularity feature unit pool that match the feature unit quota of each feature vector and fuse them into a new feature vector;

[0014] A multimodal quality inspection model is constructed to generate full-scenario quality inspection results based on the feature vectors generated by the gating network.

[0015] Furthermore, feature unit quotas are assigned to each feature vector based on scene prompts, task prompts, and confidence vectors, including:

[0016] Based on scene prompts and task prompts, a bilinear transformation method is used to generate four-dimensional initial weights;

[0017] Calculate the instance confidence of each feature vector to form a confidence vector;

[0018] The initial multidimensional weights are multiplied element-wise by the confidence vector, and then normalized to obtain a four-dimensional weight vector.

[0019] Each element in the four-dimensional weight vector is mapped to the feature unit quota of the corresponding feature vector.

[0020] Furthermore, it also includes:

[0021] The alignment matrix is ​​calculated based on the correlation between the output feature vectors of the gated network.

[0022] Based on the alignment matrix, using the original output value of the last fully connected layer of the multimodal quality inspection model as the target, backpropagation is used to calculate its gradient with respect to each feature vector, thus obtaining the percentage contribution of each gated network output feature vector to the overall quality inspection results.

[0023] Extract feature vectors whose contribution percentage is greater than a set threshold, and visualize them according to their modalities.

[0024] If no feature vector has a contribution percentage greater than the set threshold, then the multimodal shared responsibility is displayed and a weighted contribution bar is given.

[0025] Furthermore, scene prompts and task prompts are determined based on semantic feature vectors and third-person visual feature vectors, combined with prior knowledge text.

[0026] Furthermore, the first feature branch extracts the first visual feature vector using the following method:

[0027] The hand position is determined by key point detection method, and local features are extracted by combining the hand RoI.

[0028] An instance segmentation method is used to identify the edge contours of objects in a scene.

[0029] The connection between the hand and the object is extracted by spatiotemporal interaction graph, and then a set number of hand-object interaction feature units are extracted.

[0030] Based on the first contact between the pixel change recognition tool of objects between adjacent frames and the part, a set number of object-to-object interaction feature units are extracted according to the recognition results;

[0031] The extracted hand-object interaction feature units and object-object interaction feature units are concatenated to form the first visual feature vector.

[0032] Furthermore, the instance confidence of the first visual feature vector is calculated based on the average dispersion of the edges in the spatiotemporal interaction graph.

[0033] Furthermore, the second feature branch extracts the second visual feature vector using the following method:

[0034] The continuous trajectory of the human body's center coordinates is extracted based on the target tracking algorithm and divided into a human body trajectory sequence according to a set time window length;

[0035] Based on human trajectory sequences, a set number of trajectory feature units representing trajectory changes are extracted;

[0036] The skeleton key point detection method is used to extract human motion key points, and a set number of posture feature units representing the periodic changes in human posture are extracted.

[0037] The extracted trajectory feature units and pose feature units are concatenated to form a second visual feature vector.

[0038] Furthermore, the instance confidence of the second visual feature vector is calculated based on the occlusion ratio of the detected skeleton joints.

[0039] Furthermore, the third feature branch extracts the third visual feature vector using the following method:

[0040] The last layer feature map is downsampled into a mesh map containing a set number of mesh cells;

[0041] Scene layout features are extracted from each grid cell to obtain scene layout feature cells;

[0042] An instance segmentation method is used to identify specific instances in a scene, and the scene semantic feature unit corresponding to each instance is obtained.

[0043] Evaluate the global attributes of the scene to obtain global state feature units;

[0044] The scene layout feature unit, scene semantic feature unit, and global state feature unit are concatenated to form a third visual feature vector.

[0045] Furthermore, the instance confidence of the third visual feature vector is calculated based on the mask edge blur during instance segmentation.

[0046] Furthermore, the audio feature vector includes acoustic feature vector and semantic feature vector. The instance confidence of the audio feature vector is calculated based on the audio signal-to-noise ratio and the average word-level confidence obtained when extracting the semantic feature vector.

[0047] Beneficial effects

[0048] By adopting the above-mentioned technical solution, the present invention has the following advantages and positive effects compared with the prior art:

[0049] This invention decouples the traditional binary features of action and environment into multi-granularity feature units by constructing a multi-granularity feature pool, and all visual features share the same feature extraction backbone, thus avoiding the repetitive development of the traditional silo architecture.

[0050] This invention adaptively allocates weights through a three-level gating network based on scenario-task-instance confidence, achieving a comprehensive decision-making process that considers both theoretical requirements and actual data quality. Compared to existing static fusion schemes, this significantly improves the quality inspection accuracy in complex scenarios.

[0051] This invention quantifies feature contributions through alignment matrix and gradient backpropagation, solving the problem of uninterpretable decision generation in existing solutions. It enables quality inspectors to intuitively locate which action or statement caused the violation and, through counterfactual analysis, clearly identify the most effective correction method, thereby greatly improving the efficiency of trainers' debriefing. Attached Figure Description

[0052] Figure 1 This is a flowchart of an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of cross-scenario unified multi-granularity coding according to an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of feature sparsity based on a three-level adaptive gating network according to an embodiment of the present invention;

[0055] Figure 4 This is a causal attribution visualization diagram of an embodiment of the present invention. Detailed Implementation

[0056] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0057] The embodiments of the present invention relate to a full-scenario quality inspection method for automobile stores based on multi-granularity feature adaptive fusion, such as... Figure 1 As shown, it includes:

[0058] Acquire multimodal data streams;

[0059] We design a unified feature extraction backbone based on Vision Transformer (ViT) to decouple video stream data into micro-interaction, macro-body and environmental state features;

[0060] Extracting auditory dual-channel features;

[0061] Build a multi-granularity token pool;

[0062] Design a three-level adaptive gating network that dynamically allocates the fusion weights of each token subset by combining scenario priors, task priors, and instance confidence.

[0063] The features output by the gating network are input into the full-scene quality inspection model to obtain the quality inspection results;

[0064] Create attribution maps to pinpoint the contribution of the results and visualize them.

[0065] The multimodal data stream includes:

[0066] (1) Video stream (V): from showroom surveillance cameras (covering sales reception and test drive explanation scenarios), repair bay cameras (monitoring repair operation processes) and salesperson handheld devices (such as mobile phones, used for close-up recording of vehicle delivery, contract explanation, and other scenarios);

[0067] (2) Audio stream (A): Separated from the video stream (such as synchronous recording of showroom monitoring) or independently collected through a mobile phone microphone (such as one-on-one communication between a sales consultant and a customer) to extract voice content and tone of voice.

[0068] (3) Text Flow (T): Contains enterprise-level prior knowledge, such as standard sales script library (which stipulates keywords such as "must mention warranty policy"), maintenance operation manual (which defines specifications such as "tools put away" and "parts cleaning"), and fault diagnosis knowledge base (which marks abnormal features such as "oil leakage" and "loose screws"), providing rule benchmarks for quality inspection.

[0069] (4) Semantic prompting stream (P): It is composed of "scene prompt words" (such as "sales", "repair", "delivery") and "task prompt words" (such as "script compliance check" and "repair operation standardization assessment"), which serve as the prior input of the subsequent gating network and guide the tendency of feature selection. For example, the sales scenario focuses on "person-action" features, and the repair scenario focuses on "environment-tool" features.

[0070] Traditional visual feature extraction typically employs a one-size-fits-all approach (such as directly outputting all image features), failing to differentiate the varying needs of different scenarios regarding micro-interactions, macro-actions, and environmental states. For example... Figure 2 As shown, this implementation method, based on the last layer feature map of ViT (VisionTransformer) (which has multi-scale and local-global correlation characteristics), extracts three branches in parallel to extract three types of visual tokens: fine-grained interaction, macroscopic body trajectory, and environmental state semantics, achieving a single backbone and multi-granularity output. Specifically, as follows:

[0071] (1) Fine-grained interaction token (Fv_micro): Captures precise interactions between people and objects, tools and parts.

[0072] Fine-grained interaction tokens focus on the micro-operations most prone to violations in automotive store scenarios (such as whether the sales consultant points to the warranty manual or whether the repair technician correctly handles the parts), and are extracted through the following two sub-features:

[0073] A. Hand-Object Interaction Tokens (12 tokens):

[0074] Keypoint detection (e.g., OpenPose detecting 21 hand joints) is used to locate the hand position, and local features are extracted by combining the hand's RoI. Furthermore, the validity of the interaction (e.g., "pointing" or "holding") is determined by analyzing the spatiotemporal interaction graph (analyzing the relationship between the hand's motion trajectory and the object's position over three consecutive frames). Finally, 12 tokens (12×128 dimensions) strongly correlated with hand-object interactions are extracted from the ViT feature map.

[0075] B. Tool-Part Contact Tokens (8 tokens):

[0076] The edge contours of tools (such as wrenches) and parts (such as screws) are extracted through instance segmentation (e.g., Mask R-CNN). Anomalies in pixel motion vectors between adjacent frames are calculated using optical flow mutation detection to identify contact moments, such as the first contact between the tool and the part. Finally, eight tokens (8×128 dimensions) that are strongly correlated with tool-part contact are extracted.

[0077] The above 20 tokens (20×128) focus on micro-interaction details and are suitable for fine-grained quality inspection needs such as "gesture guidance" in sales scenarios and "tool usage specifications" in repair scenarios.

[0078] (2) Macroscopic body trajectory token (Fv_macro): captures the overall coherence of human posture.

[0079] Macro-body trajectory tokens focus on the movement patterns of the human body over a longer time scale (such as whether a sales consultant's walking trajectory conforms to the reception process, or whether a repair technician's bending / turning frequency is abnormal), and are extracted through the following two sub-features:

[0080] A. Human body center trajectory tokens (5 tokens):

[0081] The continuous trajectory of the human body's center coordinates is extracted based on the target tracking algorithm and divided into 5 trajectory sequences according to the time window. Each segment can contain features such as the rate of change of coordinates and the number of directional turns, corresponding to 5 trajectory tokens (dimension 5×128).

[0082] B. Skeleton-Posture Cycle Tokens (8 tokens):

[0083] By detecting key points of the skeleton, key points of human movement (head, shoulders, knees, etc.) are extracted, and periodic changes in posture are analyzed (such as the "gesture-pause" cycle during sales presentations and the "bending-standing" cycle during repairs). Eight periodic feature tokens (dimension 8×128) are extracted.

[0084] The above totals 13 tokens (13×128), applicable to macro-level quality inspection needs such as "service process integrity" (e.g., whether the customer is accompanied throughout the process) in sales scenarios and "operational standardization" (e.g., whether any inspection steps are omitted) in repair scenarios.

[0085] (3) Environment State Semantic Token (Fv_state): Captures the global static and dynamic states of the scene.

[0086] Environmental state semantic tokens focus on non-human objects and the overall environment in a scene (such as the placement of tools in a repair shop or the cleanliness of the showroom floor), and are extracted through the following three sub-features:

[0087] A. Scene layout tokens (64 tokens):

[0088] The feature map is downsampled into an 8×8 grid (64 grid cells in total). Scene layout features (such as the spatial distribution of "reception area", "showroom area" and "repair area") are extracted from each cell, corresponding to 64 layout tokens (dimension 64×128).

[0089] B. Instance Semantic Tokens (N tokens, dynamically adjusted by dimension):

[0090] Identify tools (N_tool), parts (N_part), and surfaces (N_surface, such as car paint or work surface) in the scene. Each instance outputs one semantic token (such as "wrench", "screw", "cleaning surface"), with a total of N_tool + N_part + N_surface (dimension N×128).

[0091] C. Global State Token (1 token):

[0092] Evaluate global attributes of the scene, such as ground cleanliness, presence of oil stains, debris, etc., and output a global state token (dimension 1×128).

[0093] The above totals (64+N+1) tokens. The dimension of this vector is dynamically adjusted according to the number of instances. It is suitable for environmental standard quality inspection in maintenance scenarios such as "tool placement" and "parts arrangement", as well as for requirements such as "environmental cleanliness" in showroom scenarios.

[0094] The audio stream (A) can be decomposed into acoustic features (emotion / tone) and semantic features (keywords / script) through a dual-channel approach, corresponding to the assessment of service attitude and script compliance, respectively:

[0095] (1) Acoustic features (Fa_acoustic, 128 dimensions):

[0096] By using intonation recognition (such as extracting fundamental frequency, intensity, and speech rate) and sentiment analysis (such as classifying “friendly” and “impatient”), continuous audio frames are converted into 128-dimensional sentiment vectors.

[0097] (2) Semantic features (Fa_semantic, 256 dimensions):

[0098] Audio is converted into text using ASR (Automatic Speech Recognition) and 256-dimensional keyword vectors are extracted using word embedding.

[0099] The multi-granularity token pool is finally integrated into Φ={Fv_micro(20×128),Fv_macro(13×128),Fv_state((64+N+1)×128),Fa_acoustic(128),Fa_semantic(256)}, with a total of about 120 tokens (N is the number of dynamic instances). All tokens are unified to 128 dimensions (Fa_semantic is linearly projected to 128 dimensions).

[0100] The 3-Level Adaptive Gating (3L-AG) network is the core mechanism used in this implementation to solve the rigidity of cross-scene feature fusion. Its core idea is to dynamically adjust the fusion weight of various features in the multi-granularity token pool by combining scene priors, task requirements and real-time data quality, so as to avoid the problem that traditional static fusion cannot adapt to scene differences.

[0101] like Figure 3 As shown, the three layers of input information in the gating network correspond to scenario type (e.g., sales / repair), specific task (e.g., script review / tool ​​return), and instance confidence, respectively. These three layers of information are progressively refined from macro-level scenario requirements to micro-level data status, and together determine the dynamic weights of feature fusion.

[0102] The scene-level prior E_scene is used to define scene feature tendencies, and its input includes scene cue words in the semantic cue stream P.

[0103] Scene cues are first converted into vectors through text embedding, then input into a two-layer MLP for nonlinear transformation, outputting a 16-dimensional vector e_s. e_s implicitly contains the scene's natural preference for features. For example, e_s in a sales scene encodes prior knowledge of "needing to pay attention to hand movements (Fv_micro) and speech semantics (Fa_semantic)"; while e_s in a repair scene encodes prior knowledge of "needing to pay attention to tool state (Fv_state) and macroscopic trajectory (Fv_macro)".

[0104] The task-level prior E_task is used to refine the task feature requirements. Its input includes task prompt words from the semantic prompt stream P (such as "script compliance check" and "maintenance operation standardization assessment"). The task prompt words are inserted into the pre-trained language model (such as BERT) using LoRA (Low-Rank Adaptation), and only a few parameters need to be fine-tuned to output a 16-dimensional vector e_t. The advantage of LoRA is that it approximates full parameter fine-tuning through a low-rank matrix, reducing computational cost while preserving task specificity.

[0105] e_t further refines the characteristic requirements of the scenario. For example, in the same sales scenario, the e_t of the "script compliance" task will strengthen the weight of audio semantics (Fa_semantic), while the e_t of the "service attitude assessment" task will strengthen the weight of acoustic emotion (Fa_acoustic).

[0106] The instance-level confidence score Q_t is used to evaluate the quality of real-time data. Its input includes various tokens (Fv_micro, Fv_macro, Fv_state, Fa_acoustic / Fa_semantic) in the multi-granularity token pool.

[0107] Calculate the reliability of the current instance for each subset of tokens and output a 4-dimensional confidence vector. Each dimension corresponds to a type of token, specifically including:

[0108] (1) Micro-interaction confidence Q_v_micro: 1−(mean dispersion of the edges of the hand-object interaction graph):

[0109] Hand-object interaction graphs can be modeled using graph neural networks (GNNs) to represent the connections between hand keypoints and objects. The dispersion of edges reflects the stability of the interaction (the greater the dispersion, the more severe the hand tremors or occlusion). The closer Q_v_micro is to 1, the clearer the micro-interaction features; the closer it is to 0, the more ambiguous the interaction.

[0110] (2) Macroscopic trajectory confidence Q_v_macro: 1− (skeleton occlusion rate):

[0111] Skeletal occlusion rate is calculated using skeleton keypoint detection (such as HRNet) to determine the proportion of occluded joints (e.g., the occlusion rate increases when the head is occluded). The closer Q_v_macro is to 1, the more complete the body trajectory; the closer it is to 0, the more fragmented the trajectory.

[0112] (3) Environmental confidence Q_v_state: 1−(segmentation mask edge ambiguity):

[0113] The blurriness of the segmentation mask edge is calculated by measuring the gradient strength of the segmentation edge (blurred edges have low gradients). The closer Q_v_state is to 1, the more defined the environment state; the closer it is to 0, the more blurred the environment.

[0114] (4) Audio confidence Q_a: 0.4 × Q_acoustic + 0.6 × Q_semantic:

[0115] in:

[0116] Q_acoustic = 1 / (1 + exp(15 - SNR)), where SNR is the signal-to-noise ratio (dB) of the entire audio segment, calculated through the speech activity detection (VAD) region;

[0117] Q_semantic = mean(ASR_word_confidences), which is the mean word-level confidence score output by ASR.

[0118] This design enables the system to assess audio reliability through acoustic quality even in silent text scenarios (such as pure ambient sound).

[0119] Q_t quantifies the availability of various tokens in the current data, avoiding model reliance on unreliable features.

[0120] The scenario-level prior e_s, task-level prior e_t, and instance-level confidence Q_t are fused through the following steps to generate 4-dimensional dynamic weights [w_micro,w_macro,w_state,w_audio]:

[0121] (1) Bilinear gating enables the fusion of prior knowledge:

[0122] The scene prior e_s (16-dimensional) and the task prior e_t (16-dimensional) are combined by bilinear transformation to generate 4-dimensional prior weights. ,W_prior=Bilinear(e_s,e_t)=e_s T ·W·e_t.

[0123] Here, W is a 16×16 learnable matrix that can be obtained through training and optimization. The advantage of bilinear transformation is that it can capture the non-linear interaction between e_s and e_t. For example, when e_s in a sales scenario is combined with e_t in a "script compliance" task, it will strengthen the weight of audio semantics.

[0124] (2) Confidence calibration achieves a balance between prior knowledge and data quality:

[0125] The prior weights W_prior reflect the features that should theoretically be of interest, but need to be further calibrated in conjunction with the instance confidence Q_t, i.e., the reliable features that can actually be obtained. The final weights W_final are obtained by multiplying Q_t element-wise after Softmax normalization: W_final = Softmax(W_prior ⊙ Q_t), where ⊙ represents element-wise multiplication. Softmax ensures that the sum of the weights is 1, while Q_t performs reliability filtering on the prior weights. For example, if Q_a = 0.2, even if the audio weights in W_prior are high, the final W_audio will be lowered.

[0126] After obtaining the 4-dimensional dynamic weight W_final, it needs to be mapped to a specific number of tokens to achieve token-level sparse selection (retaining only the tokens most useful for the current task). The specific rules are as follows:

[0127] (1) Fine-grained interaction token (Fv_micro): Select Each Token (20 is the total number of tokens in Fv_micro). For example, when w_micro=0.52, That is, retain the 11 most relevant fine-grained interaction tokens;

[0128] (2) Macroscopic body trajectory Token (Fv_macro): Select Each Token (13 is the total number of tokens in Fv_macro). For example, when w_macro=0.25, That is, retain the 4 tokens that best reflect the body's trajectory;

[0129] (3) Environment State Semantic Token (Fv_state): Selection Each Token (64 + N + 1 is the total number of tokens in Fv_state, where N is the number of instances). For example, if w_state = 0.13 and N = 10, then... That is, retain the 10 tokens that best reflect the state of the environment;

[0130] (4) Auditory dual-channel token (Fa_acoustic / Fa_semantic): The contribution to the final fusion result is directly determined by the feature unit quota w_audio, that is, it is multiplied by the scaling factor w_audio during fusion. For example, when w_audio=0.10, the contribution of audio features is compressed to 10%, and the system mainly relies on visual features for decision-making.

[0131] After sparse selection, only the selected tokens enter the downstream fusion Transformer, which greatly reduces the amount of computation.

[0132] like Figure 4 As shown, to address the black-box problem, a causal attribution visualization mechanism is designed to pinpoint the specific source of violations, including:

[0133] (1) Cross-modal alignment and attribution graph generation

[0134] The alignment matrix A_align is obtained by calculating the intermodal correlation through the cross-attention mechanism;

[0135] Grad-CAM++ is used to send back the final classification logit to generate an attribution weight graph G for each token, quantifying its contribution to violation decision-making.

[0136] (2) Set attribution decision visualization rules

[0137] If audio semantic token contribution →Illegal keywords (such as "no warranty") are highlighted in red on the interface.

[0138] If micro-interaction token contribution → Overlay a green heatmap of "hand-object" onto the video frame (if it does not point to the hand area in the warranty manual);

[0139] Otherwise → Display "Multimodal Shared Responsibility" and provide a weight contribution bar for each token.

[0140] (3) Counterfactual heatmap

[0141] By using counterfactual analysis (the confidence decrease value ΔConf after removing a subset of tokens), a blue-green-red gradient heatmap is used to show that the larger ΔConf is, the greater the impact of the token on the result. For example, it helps trainers see "which action / phrase is most effective to correct".

[0142] (4) Training strategies

[0143] To enhance the adaptive capability of the gating network, data augmentation is introduced during the fine-tuning phase:

[0144] Audio is dropped in whole frames with a probability of 0.15 to simulate a noisy environment;

[0145] With a probability of 0.15, randomly mask 50% of the hand token to simulate occlusion;

[0146] Discard all environment tokens with a probability of 0.1 to simulate low light conditions.

[0147] By employing the above strategy, the gating network is forced to learn to compensate using tokens from other dimensions when the confidence of a certain type of token drops sharply, ultimately achieving a truly adaptive weight allocation.

[0148] Taking a noisy showroom scenario as an example, during the morning peak of the auto show, the background noise level was 75dB, the audio confidence level Q_a=0.21, and the gating network output weights w_micro=0.52, w_macro=0.25, w_state=0.10, and w_audio=0.13. The system selected 10 "gesture-car key" tokens (Fv_micro) and 3 "trajectory" tokens (Fv_macro), discarding the audio tokens. Quality inspection results showed that the sales consultant's failure to point to the warranty manual was detected. The attribution plot indicated that 82% of the responsibility stemmed from Fv_micro, and the trainer quickly located the problem by highlighting the hand area.

[0149] Taking a dimly lit workshop scenario at night (illuminance < 50 lx) as an example, the micro-interaction confidence Q_v_micro = 0.18, the environment confidence Q_v_state = 0.73, and the gating network automatically increases w_state to 0.65 and decreases w_micro to 0.09. The system relies on environmental state tokens such as "tools returned to their place" and "floor cleanliness" to complete quality inspection, maintaining an accuracy of 92%.

Claims

1. A method for full-scenario quality inspection of automobile dealerships based on multi-granularity feature adaptive fusion, characterized in that, include: Acquire video and audio streams collected within the car dealership; A first feature extraction network is constructed to extract multi-scale visual features from the acquired video stream, and multiple feature branches are extracted in parallel on its last layer feature map, including a first feature branch for capturing fine-grained interactions, a second feature branch for capturing macroscopic body trajectories, and a third feature branch for capturing environmental state semantics. A second feature extraction network is constructed to extract audio feature vectors from the acquired audio stream; The first visual feature vector generated by the first feature branch, the second visual feature vector generated by the second feature branch, the third visual feature vector generated by the third feature branch, and the feature units contained in the audio feature vector are collected to form a multi-granularity feature unit pool. The instance confidence of the first visual feature vector, the second visual feature vector, the third visual feature vector, and the audio feature vector are calculated respectively to quantify the real-time reliability of each feature vector in the actual scene and form a confidence vector. The process involves determining scene and task prompts, constructing a gating network to allocate feature unit quotas to each feature vector based on these prompts and the confidence vector. Then, a pool of feature units matching the quotas of each feature vector is selected from a multi-granularity feature unit pool and fused together to form a new feature vector. The allocation of feature unit quotas to each feature vector based on these prompts includes: generating four-dimensional initial weights using a bilinear transformation based on the scene and task prompts; multiplying the initial weights element-wise with the confidence vector and normalizing the result to obtain a four-dimensional weight vector; and mapping each element of the four-dimensional weight vector to the corresponding feature unit quota of the feature vector. A multimodal quality inspection model is constructed to generate full-scenario quality inspection results based on the feature vectors generated by the gating network.

2. The full-scenario quality inspection method according to claim 1, characterized in that, Also includes: The alignment matrix is ​​calculated based on the correlation between the output feature vectors of the gated network. Based on the alignment matrix, using the original output value of the last fully connected layer of the multimodal quality inspection model as the target, backpropagation is used to calculate its gradient with respect to each feature vector, thus obtaining the percentage contribution of each gated network output feature vector to the overall quality inspection results. Extract feature vectors whose contribution percentage is greater than a set threshold, and visualize them according to their modalities. If no feature vector has a contribution percentage greater than the set threshold, then the multimodal shared responsibility is displayed and a weighted contribution bar is given.

3. The full-scenario quality inspection method according to claim 1, characterized in that, The first feature branch extracts the first visual feature vector using the following method: The hand position is determined by key point detection method, and local features are extracted by combining the hand RoI. An instance segmentation method is used to identify the edge contours of objects in a scene. The connection between the hand and the object is extracted by spatiotemporal interaction graph, and then a set number of hand-object interaction feature units are extracted. Based on the first contact between the pixel change recognition tool of objects between adjacent frames and the part, a set number of object-to-object interaction feature units are extracted according to the recognition results; The extracted hand-object interaction feature units and object-object interaction feature units are concatenated to form the first visual feature vector.

4. The full-scenario quality inspection method according to claim 3, characterized in that, The instance confidence of the first visual feature vector is calculated based on the average dispersion of the edges in the spatiotemporal interaction graph.

5. The full-scenario quality inspection method according to claim 1, characterized in that, The second feature branch extracts the second visual feature vector using the following method: The continuous trajectory of the human body's center coordinates is extracted based on the target tracking algorithm and divided into a human body trajectory sequence according to a set time window length; Based on human trajectory sequences, a set number of trajectory feature units representing trajectory changes are extracted; The skeleton key point detection method is used to extract human motion key points, and a set number of posture feature units representing the periodic changes in human posture are extracted. The extracted trajectory feature units and pose feature units are concatenated to form a second visual feature vector.

6. The full-scenario quality inspection method according to claim 5, characterized in that, The instance confidence of the second visual feature vector is calculated based on the occlusion ratio of the detected skeleton joints.

7. The full-scenario quality inspection method according to claim 1, characterized in that, The third feature branch extracts the third visual feature vector using the following method: The last layer feature map is downsampled into a mesh map containing a set number of mesh cells; Scene layout features are extracted from each grid cell to obtain scene layout feature cells; An instance segmentation method is used to identify specific instances in a scene, and the scene semantic feature unit corresponding to each instance is obtained. Evaluate the global attributes of the scene to obtain global state feature units; The scene layout feature unit, scene semantic feature unit, and global state feature unit are concatenated to form a third visual feature vector.

8. The full-scenario quality inspection method according to claim 7, characterized in that, The instance confidence of the third visual feature vector is calculated based on the mask edge blur during instance segmentation.

9. The full-scenario quality inspection method according to claim 1, characterized in that, Audio feature vectors include acoustic feature vectors and semantic feature vectors. The instance confidence of audio feature vectors is calculated based on the audio signal-to-noise ratio and the average word-level confidence obtained when extracting semantic feature vectors.

Images