Method, system, medium and product for visual recognition fusing large model and visual model

Abstract

A method, system, medium and product for visual recognition by fusing a large model and a visual model, relate to the technical field of visual recognition. The method comprises: obtaining a preset number of initial sample images of a target recognition category, inputting the initial sample images into a target multi-modal large model, generating a fine-grained structured text description corresponding to each initial sample image, and constructing an enhanced data set; training a target visual model using the enhanced data set; in response to receiving a new input image, obtaining a first category probability distribution output by the target visual model in parallel, and a second category probability distribution output by the target multi-modal large model based on semantic similarity matching, and determining a final recognition result according to the first category probability distribution and the second category probability distribution. The above technical solution can improve the recognition accuracy in special visual recognition scenarios in the non-structured visual recognition scene with few samples.

Description

Technical Field

[0001] This application relates to the field of visual recognition, and more particularly to a method, system, medium, and product for visual recognition that integrates large models and visual models. Background Technology

[0002] In today's era of rapid technological development, the field of computer vision has made great strides. Visual recognition technology, as a key component, has been widely applied in many fields and plays a vital role.

[0003] Traditional visual recognition methods primarily employ a single visual model for image recognition. A common approach is based on feature extraction and classifiers. This involves first extracting specific features from the image, such as color, texture, and shape, and then using a classifier to categorize these features to determine the class of the target in the image. Another approach is based on deep learning convolutional neural networks, which train the network on a large amount of image data, allowing it to automatically learn image features and patterns. These methods can achieve certain results when trained on large-scale datasets.

[0004] However, existing visual recognition technologies have certain shortcomings. A single visual model often falls short of the required accuracy when faced with complex scenes and diverse targets. This is especially true for unstructured visual recognition scenarios with few samples, such as identifying special or rare objects (e.g., specific types of factory vehicles). Traditional model training faces challenges because they lack a deep understanding of the semantic information of the images, resulting in poor recognition performance and low reliability of the results. Summary of the Invention

[0005] This application provides a method, system, medium, and product for visual recognition that integrates large models and visual models, which can improve the recognition accuracy in special visual recognition scenarios with few samples and unstructured visual recognition scenarios.

[0006] In a first aspect, this application provides a method for visual recognition that integrates large-scale models and visual models, the method comprising:

[0007] Textual information from the target domain is collected to form a domain knowledge set, which includes structured knowledge text and unstructured descriptive text. The domain knowledge set and the corresponding domain image are used as training pairs to perform comparative learning on a pre-trained visual-language multimodal large model, and an elastic weight consolidation strategy is adopted simultaneously to obtain the target multimodal large model.

[0008] A predetermined number of initial sample images for the target recognition category are obtained. The initial sample images are input into the target multimodal large model to generate a fine-grained structured text description corresponding to each initial sample image, resulting in an initial sample-description pair. The fine-grained structured text description includes object category, component composition, attribute state, and scene context. Based on the initial sample-description pair, a synthetic image-combined description pair is generated through semantic decoupling and recombination. An augmented dataset is constructed based on the initial sample-description pair and the synthetic image-combined description pair.

[0009] The initial visual model is trained using the augmented dataset by minimizing a collaborative loss function to obtain a target visual model. The collaborative loss function includes a first loss, a second loss, and a third loss. The first loss is the standard classification loss of the initial visual model calculated based on the category labels of the initial sample images in the augmented dataset. The second loss is the distance between the visual features extracted by the initial visual model from the initial sample images in the augmented dataset and the text features extracted by the target multimodal large model from the fine-grained structured text description corresponding to the augmented dataset. The third loss is the difference between the relative relationships between categories in the feature space of the initial visual model and the domain cognitive graph constructed by the target multimodal large model.

[0010] In response to receiving a new input image, the system acquires in parallel the first category probability distribution output by the target visual model and the second category probability distribution output by the target multimodal large model based on semantic similarity matching. The system then fuses the first category probability distribution and the second category probability distribution using a preset evidence theory to obtain the fusion confidence of each candidate category. If the highest fusion confidence is higher than or equal to the current dynamic confidence threshold, the category corresponding to the highest fusion confidence is output as the final recognition result.

[0011] By employing the aforementioned technical solution, through contrastive learning and elastic weight consolidation, structured and unstructured knowledge from the target domain is safely and effectively integrated into a multimodal large model, enabling it to possess deep semantic understanding and reasoning capabilities specific to the domain, thus laying a knowledge foundation for subsequent processing. Utilizing the domain-enhanced large model, fine-grained structured semantic descriptions are automatically generated from a very small number of initial images. Based on these descriptions, semantic decoupling and reorganization are performed to create diverse synthetic training data, significantly alleviating the core bottleneck of scarce and insufficient training data diversity under limited sample conditions. A collaborative loss function is designed to not only optimize classification accuracy during visual model training but also force its visual features to align with the deep textual semantic features of the multimodal large model, and ensure that the learned category relationships are consistent with the domain cognitive graph. This transfers and solidifies the domain knowledge of the large model into a lightweight visual model, improving its generalization and discriminative abilities. During the reasoning stage, the discrimination results of the dedicated visual model and the semantic matching results of the knowledge-based large model are combined in parallel, and the uncertainty of the two is quantified and integrated using evidence theory to achieve complementary advantages and ultimately improve the system's robustness, interpretability and overall accuracy in complex and uncertain scenarios.

[0012] In some embodiments, the step of using the domain knowledge set and the corresponding domain image as training pairs to perform comparative learning on a pre-trained visual-language multimodal large model, and simultaneously employing an elastic weight consolidation strategy to obtain the target multimodal large model, specifically includes:

[0013] A batch of training pairs is selected. For the first training pair in the batch, the first optimization objective is to maximize the cosine similarity between image features and text features in the cross-modal embedding space. The second optimization objective is to minimize the cosine similarity between the image features of the first training pair and other text features in the batch, as well as the cosine similarity between the text features of the first training pair and other image features in the batch. The first training pair can be any training pair in the batch.

[0014] The contrastive learning loss function is calculated based on the first optimization objective and the second optimization objective. The Fisher information matrix of each parameter of the visual-language multimodal large model is evaluated based on all training pairs, and the diagonal elements of the Fisher information matrix are taken to form the parameter importance vector.

[0015] The regularization term is obtained based on the training difference between each element in the parameter importance vector and the corresponding parameter of the visual-language multimodal large model, and the contrastive learning loss function is added to the regularization term to construct a joint training loss function;

[0016] With the goal of minimizing the joint training loss function, the vision-language multimodal large model is iteratively optimized, and the converged model is taken as the target multimodal large model.

[0017] By employing the aforementioned technical solution, through contrastive learning, images within the target domain are forced to exhibit a high degree of similarity in feature space to their specific descriptive text, while avoiding other mismatched image or text features. This ensures that the visual concepts learned by the model establish accurate and robust semantic associations with domain textual knowledge, laying the foundation for domain specialization in large models. The simultaneously employed elastic weight consolidation strategy is crucial; it identifies and quantifies the parameter importance of general knowledge learned in the pre-trained large model using the Fisher information matrix. When optimizing for new domains, these important parameters are penalized, effectively constraining their variation, thereby efficiently absorbing new domain knowledge while maximizing the retention of the model's original powerful general understanding and reasoning capabilities. This training process is not merely a simple feature mapping. By conducting large-scale, refined contrastive learning between the domain knowledge set (containing structured and unstructured text) and image pairs, the model essentially constructs an implicit "cognitive map" of the domain, linking visual and textual aspects. This provides a deep knowledge framework for subsequent generation of fine-grained descriptions and semantic reasoning.

[0018] In some embodiments, the step of evaluating the Fisher information matrix of each parameter of the visual-language multimodal large model based on all training pairs, and taking the diagonal elements of the Fisher information matrix to form a parameter importance vector, specifically includes:

[0019] For the second training pair among all training pairs, calculate the gradient of the contrastive learning loss function with respect to the model parameters, and record the gradient components of the parameters involved in the forward propagation paths of the image modality and the text modality in the second training pair respectively. The second training pair is any one of the training pairs among all training pairs.

[0020] Iterate through all training pairs, and aggregate the gradient components of the target parameter that appear on all image modal forward paths and all text modal forward paths respectively. Calculate the mean of the squared gradients in the image modality and the text modality respectively, and use it as an importance estimate of the target parameter in the corresponding modality. The target parameter can be any parameter.

[0021] The importance estimates of the target parameter in the image modality and the importance estimates in the text modality are weighted and summed according to preset fusion weights to obtain the multimodal fusion importance score of the target parameter;

[0022] The multimodal fusion importance scores of all parameters are arranged in order to form an initial importance vector. The initial importance vector is then smoothed and normalized to obtain the parameter importance vector.

[0023] The above technical solution distinguishes between image and text forward propagation paths, calculating the importance of parameters under different modalities. This reflects the model's differentiated contribution to knowledge learning across different modalities, providing a basis for subsequent refined constraints. By traversing all training pairs and aggregating gradient components, the mean of the squared gradients is calculated, effectively reducing random noise based on single-sample estimations, making the parameter importance assessment more stable, reliable, and statistically significant. Innovatively, the importance estimates of image and text modalities are weighted and summed using preset fusion weights to obtain a unified multimodal fusion importance score. This allows the final importance assessment to comprehensively weigh the dual contributions of parameters to visual and language understanding, better aligning with the learning objectives of large multimodal models.

[0024] In some embodiments, generating a synthetic image-combined description pair based on the initial sample-description pair through semantic decoupling and recombination includes:

[0025] The fine-grained structured text description in each initial sample-description pair is decoupled into an independent set of semantic units using the target multimodal large model. Each semantic unit in the set of semantic units corresponds to a component, attribute, or context label.

[0026] Each semantic unit in the set of semantic units is text-encoded to obtain the corresponding unit text feature vector. The initial sample image is visually encoded, and the local visual feature vector corresponding to each semantic unit is extracted through a region attention mechanism.

[0027] From the semantic unit sets decoupled from different initial sample-description pairs, one or more semantic units are selected respectively and combined into a new semantic unit combination. According to the semantic unit combination, the corresponding local visual feature vectors are selected from each source image, and spatially perceptual feature fusion is performed to generate synthetic image features.

[0028] The synthetic image features are input into a pre-trained diffusion model or generative adversarial network to generate the synthetic image. The semantic units are combined and reconstructed into a combined text description that conforms to the grammatical structure to obtain the synthetic image-combined description pair.

[0029] By employing the aforementioned technical solution, the textual description of the initial samples is decomposed into independent, fine-grained semantic units using a target multimodal large model. Through textual and visual encoding, a precise correspondence is established between these textual semantic units and local visual regions of the image. This achieves a deep and structured understanding of the image content, laying the foundation for subsequent creative recombination. Arbitrary selection and combination of the decoupled semantic units among different samples creates entirely new semantic unit combinations. This is equivalent to data augmentation at the semantic level rather than the pixel level, generating visually novel yet semantically reasonable and diverse samples, significantly overcoming the limitations of the initial data volume. By mapping the recombined semantic unit combinations back to their source local visual features and performing spatially perceptual feature fusion, a synthetic image feature corresponding to the new semantic combination can be generated. Subsequently, a generative model is used to generate the final image. This process ensures that the generated synthetic image is strictly consistent with the combined semantic description in content, achieving controllable generation from high-order semantic concepts to low-order visual features.

[0030] In some embodiments, the second loss is calculated through the following steps:

[0031] For the first target sample in the augmented dataset, the visual features of the initial sample image are extracted using the initial visual model, and the text features of the corresponding fine-grained structured text description are extracted using the target multimodal large model. The visual features and the text features are mapped to a common semantic contrast space, respectively. The first target sample is any initial sample-description pair in the augmented dataset.

[0032] In the public semantic comparison space, the overall similarity between the mapped visual features and the mapped text features of the first target sample is calculated. The text description is decomposed into multiple semantic components, and the local text features of each semantic component are extracted. The local similarity between the mapped visual features and each of the local text features is calculated.

[0033] Based on the image quality and the level of detail of the text description of the first target sample, the contribution weights of the overall similarity and the local similarity in the calculation are dynamically adjusted, and the overall similarity and the local similarity are weighted and summed based on the contribution weights to obtain the final semantic matching degree.

[0034] An alignment loss term is constructed based on the final semantic matching degree. The value of the alignment loss term increases as the final semantic matching degree decreases. The second loss is obtained by averaging the alignment loss terms of all samples in the augmented dataset.

[0035] By employing the aforementioned technical solution, text descriptions are decomposed into multiple semantic components, and the local similarity between visual features and each local text feature is calculated. This forces the visual model not only to understand the overall concept of the image but also to learn the precise correspondence between its internal local visual regions and specific, fine-grained semantic concepts, thereby achieving a deeper and more interpretable alignment. An innovative dynamic weight adjustment mechanism is introduced, determining the contribution ratio of overall similarity to local similarity based on the image quality and the level of detail in the text description. This addresses potential issues with sample quality and inconsistent description granularity during data augmentation, making the alignment objective more robust and flexible, and enabling adaptive utilization of the most reliable information from each sample. The final semantic matching degree calculated from all samples is transformed into an alignment loss term, and averaged to obtain a stable second loss term. This continuously guides the visual model during training, causing its extracted feature space to constantly approach the domain-knowledge-rich semantic space constructed by the target multimodal large model, achieving continuous and effective distillation of knowledge from the large model to the visual model.

[0036] In some embodiments, the third loss is calculated through the following steps:

[0037] For the target category, all initial sample images belonging to the target category in the augmented dataset are collected, the visual features of the initial sample images are extracted using the initial visual model, and the visual prototype features of the target category are obtained through aggregation operations.

[0038] The visual prototype features of each of the multiple target categories are obtained, the similarity between each pair of the multiple visual prototype features is calculated, and a visual concept relationship matrix is ​​constructed. The elements in the visual concept relationship matrix represent the association strength between two categories in the visual feature space.

[0039] The names or descriptive texts of multiple target categories are input into the target multimodal large model, and the semantic relationship strength between the target categories is extracted through the target multimodal large model to construct a semantic concept relationship matrix;

[0040] The difference measure between the visual concept relation matrix and the semantic concept relation matrix is ​​calculated and used as the third loss.

[0041] By employing the aforementioned technical solution, visual prototype features are formed by aggregating visual features from similar samples, and pairwise similarities between different categories of prototypes are calculated to construct a visual concept relationship matrix. This quantifies the inter-category relationships perceived by the visual model, making the model's implicit visual knowledge explicit and structured, laying the foundation for comparison with higher-level semantic knowledge. The name or descriptive text of the target category is input into the target multimodal large model, directly utilizing the prior semantic relationships learned from massive domain knowledge inherent in the large model to construct a semantic concept relationship matrix. A third loss is constructed by calculating the difference between the two relationship matrices. This third loss mandates that the visual relationships between categories learned by the visual model must be consistent with the semantic relationships defined by the multimodal large model and conform to domain common sense. This guides the visual model to organize categories in the feature space according to correct semantic logic. This alignment ensures that the visual model does not learn each category in isolation, but rather learns within a "concept network" that conforms to the domain knowledge structure. This enables the model to understand the advanced principle that "similar categories should also be closer in the feature space," which helps the model better distinguish easily confused categories and make more reasonable inferences about samples that were not fully seen during training. This significantly improves the model's discrimination accuracy and structured generalization ability in scenarios with few samples, fine granularity, and unstructured environments.

[0042] In some embodiments, the step of fusing the probability distributions of the first category and the second category using a preset evidence theory to obtain the fusion confidence score of each candidate category specifically includes:

[0043] The first category probability distribution is transformed into a first basic confidence assignment to characterize the degree of support for each candidate category, and the second category probability distribution is transformed into a second basic confidence assignment to characterize the degree of support for each candidate category.

[0044] Calculate the conflict measure between the first basic confidence assignment and the second basic confidence assignment;

[0045] If the conflict metric is lower than a preset threshold, then the first basic confidence assignment and the second basic confidence assignment are combined according to the preset evidence theory combination rules to obtain a joint basic confidence assignment.

[0046] Based on the joint basic confidence assignment, the confidence function value and likelihood function value of each candidate category are calculated, and the fusion confidence of each candidate category is obtained by weighted averaging of the confidence function value and the likelihood function value.

[0047] By employing the aforementioned technical solution, the system can automatically identify significant contradictions between visual and semantic evidence by calculating a conflict metric between two basic confidence assignments and comparing it to a preset threshold. Standard evidence combination is only performed when the conflict is below the threshold. This mechanism provides a safety valve for the fusion process, preventing erroneous decisions caused by forced fusion when two pieces of evidence are severely contradictory, thus improving the system's robustness in marginal or anomalous situations. Under low-conflict conditions, fusion is performed according to the combination rules of evidence theory. These rules not only superimpose the support of two evidence sources for the same category but also handle mutual corroboration and partial conflicts between evidence through mathematical operations, ultimately resulting in a "joint basic confidence assignment" that integrates all available information. This is more scientific and in-depth than simple weighted averaging or voting mechanisms.

[0048] In a second aspect, embodiments of this application provide a computer system including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the steps of the method described in any possible implementation of the first aspect.

[0049] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of the method described in any possible implementation of the first aspect.

[0050] Fourthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the method described in any possible implementation of the first aspect.

[0051] It is understood that the computer system provided in the second aspect, the storage medium provided in the third aspect, and the computer program product provided in the fourth aspect are all used to execute the method provided in this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0052] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0053] 1. By utilizing a target multimodal large model to generate fine-grained text descriptions from a small number of initial samples, and combining semantic decoupling and reorganization techniques to construct a large-scale and diverse augmented dataset, the bottleneck of insufficient training data is effectively overcome, enabling the model to still achieve excellent recognition capabilities even with only a small number of labeled samples.

[0054] 2. By injecting domain knowledge (structured and unstructured text) and training with fine-grained descriptions containing information such as components, attributes, and context, the model is able to understand deep semantic information beyond simple category labels. This significantly improves the model's accuracy and robustness when facing special visual recognition tasks with ambiguous category definitions, complex structures, or those belonging to specialized domains.

[0055] 3. The designed collaborative loss function forces the visual model to align with the knowledge-rich multimodal large model at the feature level. This not only distills the semantic knowledge of the large model into the visual model, but also makes its learned category relationships conform to the domain cognitive graph. This collaborative training ensures that the lightweight visual model can inherit the powerful semantic understanding capabilities of the large model;

[0056] 4. During the reasoning stage, the prediction results of the visual model and the knowledge model are integrated using evidence theory to quantify the uncertainties and conflicts of different evidence sources, achieving more scientific and robust decision-making. Especially when the predictions of the two models are inconsistent or the input image is ambiguous, this fusion mechanism can comprehensively evaluate and output recognition results with higher confidence and stronger interpretability, thereby reducing the risk of misjudgment. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating a method for visual recognition that integrates a large model and a visual model, as described in an embodiment of this application.

[0058] Figure 2 This is a schematic diagram of the process for generating a target multimodal large model in an embodiment of this application;

[0059] Figure 3 This is a schematic diagram of an exemplary hardware structure of a computer system in an embodiment of this application. Detailed Implementation

[0060] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0061] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0062] The following is combined with Figure 1 The method of the embodiments of this application will be described below.

[0063] Please see Figure 1 This is a flowchart illustrating a method for visual recognition that integrates a large model and a visual model, as described in an embodiment of this application. Figure 1 As shown, a method for visual recognition that integrates large models and visual models includes the following steps:

[0064] S101. Collect textual information from the target domain to form a domain knowledge set. The textual information includes structured knowledge text and unstructured descriptive text. Use the domain knowledge set and the corresponding domain image as training pairs to perform comparative learning on the pre-trained visual-language multimodal large model. Simultaneously, adopt an elastic weight consolidation strategy to obtain the target multimodal large model.

[0065] S102. Obtain a preset number of initial sample images for the target recognition category, input the initial sample images into the target multimodal large model, generate a fine-grained structured text description corresponding to each initial sample image, and obtain an initial sample-description pair. The fine-grained structured text description includes object category, component composition, attribute state and scene context. Based on the initial sample-description pair, generate a synthetic image-combined description pair through semantic decoupling and recombination. Construct an augmented dataset based on the initial sample-description pair and the synthetic image-combined description pair.

[0066] S103. The initial visual model is trained using the augmented dataset by minimizing the collaborative loss function to obtain the target visual model. The collaborative loss function includes a first loss, a second loss, and a third loss. The first loss is the standard classification loss of the initial visual model calculated based on the category labels of the initial sample images in the augmented dataset. The second loss is the distance between the visual features extracted by the initial visual model from the initial sample images in the augmented dataset and the text features extracted by the target multimodal large model from the fine-grained structured text description corresponding to the augmented dataset. The third loss is the difference between the relative relationship between categories in the feature space of the initial visual model and the domain cognitive map constructed by the target multimodal large model.

[0067] S104. In response to receiving a new input image, the first category probability distribution output by the target visual model and the second category probability distribution output by the target multimodal large model based on semantic similarity matching are acquired in parallel. The first category probability distribution and the second category probability distribution are fused using a preset evidence theory to obtain the fusion confidence of each candidate category. If the highest fusion confidence is higher than or equal to the current dynamic confidence threshold, the category corresponding to the highest fusion confidence is output as the final recognition result.

[0068] For the target task (e.g., identifying specific industrial parts, specific plant species, specific works of art), systematically collect and organize professional knowledge in the relevant field. Structured knowledge text typically refers to itemized information in databases, knowledge graphs, and instruction manuals. Examples include: "Part A includes components A1 and A2," "Symptoms of fault F include cracks and darkening of color," and "Plant B flowers in summer and has serrated leaves." Unstructured descriptive text typically refers to natural language descriptions in technical reports, papers, operation manuals, and online community discussions. For example: "Wearing of this type of part usually begins with slight discoloration at the edges, accompanied by the appearance of stress lines..." For the above textual information, find or create typical domain images that best represent the description, forming one-to-one training pairs. Use these training pairs to further train a pre-trained multimodal large model (such as the CLIP and BLIP series). The core objective is to maximize the similarity of matching image-text pairs in the feature space while minimizing the similarity of mismatched image-text pairs. This allows the model to learn to closely associate the visual features of domain images with domain-specific textual concepts (such as technical terms and subtle attributes). Since the multimodal large model itself already contains a massive amount of general knowledge, to prevent "forgetting the original general capabilities" when learning new domain knowledge, an elastic weight consolidation strategy is introduced simultaneously during training. The importance of each parameter in the pre-trained model to the original task (general vision-language understanding) is evaluated. This is usually quantified by calculating the Fisher information matrix of the parameters. When training for the new domain, a regularization term is added to the loss function. The regularization term penalizes excessive modifications to important parameters, meaning that when the model absorbs new knowledge, it carefully adjusts the core parameters that are crucial to maintaining the original capabilities, while mainly adjusting relatively minor parameters to adapt to the new domain. After this step, the target multimodal large model is obtained.

[0069] For each target category to be identified, prepare only a preset number (a small number, e.g., 5-10 images) of initial sample images. Input these images into the target multimodal large model. Leveraging the model's powerful image-text understanding capabilities, generate a fine-grained structured text description for each image. This description includes components, states (attributes), and the current environment (scene context), thus forming an initial sample-description pair. Using the large model, decompose each fine-grained description into independent semantic units. For example, decouple "a silver server with scratches on its body, red indicator lights, placed in a data center rack" into units such as [object: server], [component: body, attribute: silver, state: scratched], [component: indicator light, attribute: red], and [context: data center rack]. Select different semantic units from the descriptions of different samples and combine them. For example, take "object: server" and "component: body, attribute: silver" from the description of sample A, "component: indicator light, attribute: green" from the description of sample B, and "context: office" from the description of sample C. A new semantic combination is created: "A silver server with a green indicator light, placed in an office." Text generation: The new semantic combination is reconstructed into a fluent combined text description using a language model. Image generation: The semantic combination is used to create a synthetic image that highly matches the combined description by using a generative model that incorporates textual conditions (such as a diffusion model) or by guiding generation using the correspondence between decoupled semantic units and local image features. This yields a synthetic image-combined description pair. All initial sample-description pairs and a large number of newly generated synthetic image-combined description pairs are merged to form the final augmented dataset. This dataset visually covers a large number of new samples generated due to changes in parts, attributes, and scenes, while maintaining a high semantic relevance to the task objective.

[0070] The initial visual model is typically a standard image classification network (such as ResNet or ViT). The collaborative loss function consists of three parts: the first loss, the second loss, and the third loss. The first loss is the standard classification loss (such as cross-entropy loss). This ensures the model can correctly classify the labeled initial sample images, learning basic discriminative abilities. The second loss is the cross-modal feature alignment loss. This calculates the distance (such as cosine distance) between the features extracted by the visual model from the initial sample images and the text features extracted by the larger model from the corresponding fine-grained descriptions. The goal is to minimize this distance, which is equivalent to "distilling" the deeper semantic understanding of the images (reflected in the text features) from the larger model into the visual model, forcing the visual model to learn visual features rich in semantic information. The third loss is the structured relation alignment loss. This calculates the similarity relationships between feature prototypes (class centers) of different categories in the visual model, forming a visual concept relation matrix. Using a large model, the semantic association strength between these category names or descriptions is calculated (e.g., through text embedding similarity, or based on knowledge graph association), forming a semantic concept relation matrix. This semantic concept relation matrix is ​​an explicit expression of the domain cognitive graph of this application, reflecting the model's internal structured understanding of domain knowledge. The difference between the two matrices (e.g., KL divergence) is calculated. Minimizing this loss means forcing the visual model to learn inter-category relationships that conform to the domain knowledge graph structure implied by the large model (e.g., "screwdriver" and "wrench" should be visually closer than "screwdriver" and "keyboard"). The initial visual model is trained by minimizing the total loss. This process simultaneously optimizes classification accuracy, visual-semantic feature alignment, and the structural rationality of the feature space. After training, the target visual model is obtained.

[0071] Parallel Inference: For a new input image, the target visual model directly processes the image, outputting a first-class probability distribution, representing the result based on pure visual pattern recognition. The target multimodal large model performs semantic similarity matching (or direct matching with class descriptions) on the input image against all fine-grained structured text descriptions in the augmented dataset, outputting a second-class probability distribution, representing the result based on domain knowledge and overall semantic understanding of the image. Methods such as Dempster-Shafer evidence theory are used for fusion, transforming the two probability distributions into a basic confidence assignment function that simultaneously expresses "support," "uncertainty," and "conflict." The degree of conflict between the two confidence assignments is calculated; if the conflict is low (within an acceptable range), they are merged into a joint basic confidence assignment using a combination rule. This combination process effectively handles complementary and partially conflicting evidence. Based on the joint confidence assignment, a fused confidence score (e.g., the average of the trust function and the likelihood function) is calculated for each candidate class. This confidence score integrates all evidence from both models. A current dynamic confidence score threshold is set. This threshold can be adjusted based on task requirements, historical accuracy, or environmental factors. The highest fused confidence score is found among all candidate classes. If the highest fusion confidence level is greater than or equal to the current dynamic confidence level threshold, the identification result is considered reliable, and that category is output as the final identification result. If the highest confidence level is lower than the threshold, the system can output "uncertain" or request manual intervention, which is crucial for ensuring reliability in practical applications.

[0072] Figure 2 This is a schematic diagram of the process for generating a target multimodal large model in an embodiment of this application, such as... Figure 2 As shown, the step of using the domain knowledge set and the corresponding domain images as training pairs to perform comparative learning on the pre-trained visual-language multimodal large model, and simultaneously employing an elastic weight consolidation strategy to obtain the target multimodal large model, specifically includes:

[0073] S201. Extract a batch of training pairs. For the first training pair in the batch, maximize the cosine similarity between image features and text features in the cross-modal embedding space as the first optimization objective. Minimize the cosine similarity between the image features of the first training pair and other text features in the batch, and minimize the cosine similarity between the text features of the first training pair and other image features in the batch as the second optimization objective. The first training pair can be any training pair in the batch.

[0074] S202. Calculate the contrastive learning loss function based on the first optimization objective and the second optimization objective, evaluate the Fisher information matrix of each parameter of the visual-language multimodal large model based on all training pairs, and take the diagonal elements of the Fisher information matrix to form a parameter importance vector.

[0075] S203. Obtain a regularization term based on the training difference between each element in the parameter importance vector and the corresponding parameter of the visual-language multimodal large model, and add the contrastive learning loss function to the regularization term to construct a joint training loss function;

[0076] S204. With the goal of minimizing the joint training loss function, iteratively optimize the vision-language multimodal large model, and take the converged model as the target multimodal large model.

[0077] A batch of training pairs is randomly selected, with each batch containing multiple training pairs. For each image in the batch, its visual features are extracted using a visual encoder. For each text in the batch, its textual features are extracted using a text encoder. These features are mapped to a shared cross-modal embedding space. For any training pair in the batch (i.e., the first training pair): maximize the similarity between the image features and text features in the first training pair (typically using cosine similarity), minimize the similarity between the image features and all other text features in the batch, and simultaneously minimize the similarity between the text features and all other image features in the batch. Based on these objectives, a contrastive learning loss function is calculated. This loss function encourages matching image-text pairs to be close to each other in the embedding space, while keeping mismatched pairs far apart. Before starting domain training or in the early stages of training, the importance of each parameter of the model is evaluated using all domain training data. Based on all training pairs, the Fisher information matrix of each parameter of the visual-language multimodal large model is evaluated. The diagonal elements of the Fisher information matrix are extracted to form a parameter importance vector. Each value in this vector represents an estimate of the importance of the corresponding parameter to the model's performance on the domain data. A larger value indicates a greater impact of the parameter on the prediction result of the current task, meaning that more caution is needed during adjustment to prevent forgetting. During training, the difference between the current value and the initial pre-training value of each parameter is recorded. The difference of each parameter is weighted according to the element values ​​in the parameter importance vector. Parameters with high importance are given greater weight. The sum of squares of all weighted parameter differences constitutes a regularization term. The core function of this regularization term is to penalize excessive modifications to important parameters, thereby constraining the degree to which the model forgets important knowledge when learning new domain knowledge. The contrastive learning loss function is added to the regularization term to form the joint training loss function. The joint training loss function simultaneously includes two objectives: learning new knowledge (aligning domain-specific text and images) and preserving old knowledge (constraining changes in important parameters). With the goal of minimizing the joint training loss function, an optimization algorithm is used to perform multiple rounds of iterative training on the large-scale visual-language multimodal model. In each round of training, the model parameters learn domain-specific text and image alignment driven by the contrastive learning loss, while the changes in important parameters are suppressed under the constraint of the regularization term. Training stops once the joint loss function converges or reaches the preset number of training epochs. The model obtained at this point, which incorporates domain knowledge and retains general capabilities, is used as the target multimodal large model in subsequent processes.

[0078] In some embodiments, the step of evaluating the Fisher information matrix of each parameter of the visual-language multimodal large model based on all training pairs, and taking the diagonal elements of the Fisher information matrix to form a parameter importance vector, specifically includes:

[0079] For the second training pair among all training pairs, calculate the gradient of the contrastive learning loss function with respect to the model parameters, and record the gradient components of the parameters involved in the forward propagation paths of the image modality and the text modality in the second training pair respectively. The second training pair is any one of the training pairs among all training pairs.

[0080] Iterate through all training pairs, and aggregate the gradient components of the target parameter that appear on all image modal forward paths and all text modal forward paths respectively. Calculate the mean of the squared gradients in the image modality and the text modality respectively, and use it as an importance estimate of the target parameter in the corresponding modality. The target parameter can be any parameter.

[0081] The importance estimates of the target parameter in the image modality and the importance estimates in the text modality are weighted and summed according to preset fusion weights to obtain the multimodal fusion importance score of the target parameter;

[0082] The multimodal fusion importance scores of all parameters are arranged in order to form an initial importance vector. The initial importance vector is then smoothed and normalized to obtain the parameter importance vector.

[0083] Select any one sample pair from all training pairs (i.e., the second training pair), which contains a domain image and its corresponding domain text description. Input the second training pair into the target multimodal large model and perform forward propagation through both the image encoder and the text encoder to calculate the contrastive learning loss value for that sample. Calculate the gradient of the contrastive learning loss value with respect to all model parameters. During backpropagation, implement modal path separation recording: for image path gradients, only record and save the gradient components corresponding to parameters activated and used during the forward propagation for calculating image features; these parameters mainly belong to the visual encoder and related cross-modal projection layers. For text path gradients, only record and save the gradient components corresponding to parameters activated and used during the forward propagation for calculating text features; these parameters mainly belong to the text encoder and related cross-modal projection layers. This means that for a shared parameter that participates in both image processing and text processing (if it exists), its gradient will be recorded as the portion contributed by the image path and the portion contributed by the text path, respectively. Repeat the above steps for each sample pair in all training pairs to accumulate the gradient component history of each parameter under the "image modal path" and "text modal path," respectively. For any target parameter to be evaluated in the model, extract all gradient components that have appeared in the image modal paths of all samples from the historical records. Calculate the squares of these gradient values ​​and then take their average. This average is the importance estimate of the parameter for the image understanding task. The larger the mean of the squared gradients, the more significant the impact of changing this parameter on the image encoding output, and therefore the more important it is on the image side. Similarly, extract all gradient components that have appeared in the text modal paths of all samples from the historical records. Calculate the mean of their squares as the importance estimate of the target parameter for the text understanding task. For the target parameter, it now has two importance scores: one from the image side and one from the text side. Based on task requirements or prior knowledge, preset two fusion weights. Use these two fusion weights to weight and sum the two importance scores to obtain the multimodal fusion importance score of the target parameter. This reflects the comprehensive importance of the target parameter for the overall multimodal alignment task. Arrange the multimodal fusion importance scores of each parameter in the inherent order of the model parameters to form an initial importance vector. At this point, the scales of the values ​​in the vector may vary significantly, and noise may be present. Smoothing the initial vector (e.g., using a moving average or Gaussian filter) aims to reduce estimation noise caused by individual outliers or training fluctuations, making the importance assessment more stable and reliable. Normalizing the smoothed vector (e.g., scaling it to the [0, 1] interval, or making the sum of all its elements equal to 1) aims to make the importance scores of different parameters comparable and to provide a standardized weight scale for subsequent construction of the regularization term.

[0084] In some embodiments, generating a synthetic image-combined description pair based on the initial sample-description pair through semantic decoupling and recombination includes:

[0085] The fine-grained structured text description in each initial sample-description pair is decoupled into an independent set of semantic units using the target multimodal large model. Each semantic unit in the set of semantic units corresponds to a component, attribute, or context label.

[0086] Each semantic unit in the set of semantic units is text-encoded to obtain the corresponding unit text feature vector. The initial sample image is visually encoded, and the local visual feature vector corresponding to each semantic unit is extracted through a region attention mechanism.

[0087] From the semantic unit sets decoupled from different initial sample-description pairs, one or more semantic units are selected respectively and combined into a new semantic unit combination. According to the semantic unit combination, the corresponding local visual feature vectors are selected from each source image, and spatially perceptual feature fusion is performed to generate synthetic image features.

[0088] The synthetic image features are input into a pre-trained diffusion model or generative adversarial network to generate the synthetic image. The semantic units are combined and reconstructed into a combined text description that conforms to the grammatical structure to obtain the synthetic image-combined description pair.

[0089] For each initial sample-description pair, a pre-trained target multimodal large model is used to parse the fine-grained structured text description. The model decouples the description into a set of independent semantic units. Each unit corresponds to a specific semantic concept, such as component units like "turbine blade," "heat sink fin," and "indicator light"; attribute units like "silver," "cracked," and "flickering"; and context units like "in data center," "handheld operation," and "blurred background." The output is a set of semantic units, for example, {component: turbine blade, attribute: silver, component: cracked, context: test bench}. For each decoupled semantic unit, the large model's text encoder or a submodule is used to encode it into a unit text feature vector, which captures the abstract conceptual information of the semantic unit. For images from the same initial sample-description pair, the large model's visual encoder is used to encode the images holistically. Utilizing a region attention mechanism, the model "queries" the image feature map based on the text feature vector of each semantic unit, locating and focusing on the image region most relevant to that semantic meaning. Features of these activated local regions are extracted to form the local visual feature vector corresponding to each semantic unit. This establishes a precise correspondence between semantic units, unit text feature vectors, and local visual feature vectors. Semantic units are selected as needed from multiple sets of semantic units decoupled from different initial sample-description pairs. For example, {component: server chassis, attribute: black} is selected from sample A, {component: hard drive indicator light, attribute: green} from sample B, and {context: server room corridor} from sample C. The selected semantic units are combined into a new semantic unit combination that defines a previously undefined visual concept, such as "a black server chassis with a green hard drive indicator light, placed in a server room corridor." For each semantic unit in the new combination, its corresponding local visual feature vector in its original source image is traced. These local feature vectors from different source images are then fused. This process is spatially aware: the model (or a lightweight fusion network) needs to arrange the position and scale relationships of these local features at the feature level based on the common-sense spatial relationships of the semantic units (e.g., "indicator light" should be located on the front of the "chassis"), outputting a synthetic image feature that integrates all selected semantics and is structurally sound within the feature space. Synthetic image features are used as conditional input to guide a pre-trained image generation model (such as a diffusion model or generative adversarial network). This model renders a synthetic image based on the synthetic image features, ensuring a high degree of consistency between visual content and semantic composition during the generation process. The new semantic units are then combined and input into a (pre-trained) language model, which, according to grammatical rules and contextual logic, transforms them into a fluent, complete, and linguistically consistent combined text description.For example, the above combination can be transformed into: "This is a photo taken in a server room corridor. The main subject of the image is a black server, and the hard drive indicator light on its front is emitting a green light." Finally, the generated composite image is combined with the reconstructed composite text description to form a completely new, high-quality composite image-composite description pair.

[0090] In some embodiments, the second loss is calculated through the following steps:

[0091] For the first target sample in the augmented dataset, the visual features of the initial sample image are extracted using the initial visual model, and the text features of the corresponding fine-grained structured text description are extracted using the target multimodal large model. The visual features and the text features are mapped to a common semantic contrast space, respectively. The first target sample is any initial sample-description pair in the augmented dataset.

[0092] In the public semantic comparison space, the overall similarity between the mapped visual features and the mapped text features of the first target sample is calculated. The text description is decomposed into multiple semantic components, and the local text features of each semantic component are extracted. The local similarity between the mapped visual features and each of the local text features is calculated.

[0093] Based on the image quality and the level of detail of the text description of the first target sample, the contribution weights of the overall similarity and the local similarity in the calculation are dynamically adjusted, and the overall similarity and the local similarity are weighted and summed based on the contribution weights to obtain the final semantic matching degree.

[0094] An alignment loss term is constructed based on the final semantic matching degree. The value of the alignment loss term increases as the final semantic matching degree decreases. The second loss is obtained by averaging the alignment loss terms of all samples in the augmented dataset.

[0095] A sample, designated the first target sample, is selected from the augmented dataset. It is an initial sample-description pair, containing a real initial sample image and its corresponding fine-grained structured text description generated by a large model. The image from the sample is input into the initial visual model, whose visual encoder extracts a high-dimensional global visual feature vector, encoding the overall visual information of the image. The fine-grained structured text description from the sample is input into the target multimodal large model, whose text encoder extracts a high-dimensional global text feature vector, encoding the overall semantic information of the description. Since the feature spaces of the visual and text models are often not directly comparable, they need to be mapped to a shared common semantic contrast space through a projection layer (usually a simple linear layer or a small MLP). The goal is to align the image and text features from the same sample within this space. In the common semantic contrast space, the cosine similarity or dot product similarity between the mapped global visual features and global text features of the sample is calculated, measuring the semantic matching degree between the overall image and the overall text. The overall fine-grained text description is broken down into multiple semantic components according to its structure (e.g., clauses, phrases, or through a parser). For example, "a silver server with scratches and a red indicator light" is broken down into "silver," "scratched," "server," and "red indicator light." Using a target multimodal large model, each semantic component is encoded separately, resulting in multiple local text feature vectors. In the same common semantic contrast space, the similarity between the global visual features of the sample and each local text feature vector is calculated. This produces a set of similarity scores reflecting the semantic association strength between the overall image and each specific detail (part, attribute) in the description. This forces visual features to not only encompass the overall concept but also internalize the correspondence with individual details. The quality of the sample images is assessed, possibly based on their sharpness, noise level, or evaluation of their visual realism and semantic consistency. Lower quality indicates poorer reliability of the visual features. The text description detail assessment evaluates the level of detail and structure of the text description; the more concise or vague the description, the weaker the supervisory signal it provides. Based on the above evaluation results, the overall similarity weight and local similarity weight are dynamically calculated. If the image quality is high but the description is brief, visual features are tended to be relied upon more, potentially increasing the weight of the overall similarity weight, as reliable global visual features themselves provide a strong signal. If the description is very detailed and highly structured, the fine semantic guidance provided by the text is tended to be relied upon more, potentially increasing the weight of the local similarity weight to strengthen local alignment. If the image is synthetic and its quality is questionable, but the description is highly reliable, the overall similarity weight may be significantly reduced while the local similarity weight is increased, making learning rely more on the precise semantic decomposition of the text.Using dynamically adjusted weights, a weighted sum (or weighted sum) of the overall similarity and the mean (or weighted sum) of all local similarities is calculated: Final semantic match = Overall similarity weight * Overall similarity + Local similarity weight * Mean of local similarities. This score comprehensively reflects the overall semantic consistency between the image and the description across multiple granularities and weights. The ultimate goal is to maximize the final semantic match; therefore, the final semantic match is transformed into an alignment loss term, which should increase as the match decreases. A common implementation is to use a similarity-based loss function, such as: Alignment loss term = -log(σ(final semantic match)), where σ is the sigmoid function. The alignment loss term is calculated for each sample in the augmentation dataset, using all initial sample-description pairs (or a batch) used for training. The average of all loss terms is then calculated. This average is the final second loss.

[0096] In some embodiments, the third loss is calculated through the following steps:

[0097] For the target category, all initial sample images belonging to the target category in the augmented dataset are collected, the visual features of the initial sample images are extracted using the initial visual model, and the visual prototype features of the target category are obtained through aggregation operations.

[0098] The visual prototype features of each of the multiple target categories are obtained, the similarity between each pair of the multiple visual prototype features is calculated, and a visual concept relationship matrix is ​​constructed. The elements in the visual concept relationship matrix represent the association strength between two categories in the visual feature space.

[0099] The names or descriptive texts of multiple target categories are input into the target multimodal large model, and the semantic relationship strength between the target categories is extracted through the target multimodal large model to construct a semantic concept relationship matrix;

[0100] The difference measure between the visual concept relation matrix and the semantic concept relation matrix is ​​calculated and used as the third loss.

[0101] For each target category (e.g., "Type A valve", "Type B valve", "control panel"), collect all initial sample images belonging to that category from the augmented dataset (i.e., a small number of real samples, not synthetic images). Use the initial visual model to extract features from these images. Aggregate the features of all images belonging to the same category to generate a visual prototype feature representing the overall visual concept of that category. Aggregation is typically done by mean calculation, i.e., taking the average of all sample features. This prototype feature can be understood as the "visual center" or "standard image" of the category. After calculating the visual prototype features for all target categories, a set of prototype features is obtained. Calculate the similarity between each pair of these prototype features. Cosine similarity is usually used, with values ​​between [-1, 1] or [0, 1]. A larger value indicates that the visual features of the two categories are closer in space and more visually similar. For example, calculate the similarity between the prototypes of "Type A valve" and "Type B valve", and between the prototypes of "Type A valve" and "control panel". Fill a matrix with the calculated similarities between all category pairs according to a fixed category order. This matrix is ​​symmetric (because similarity is mutual), and its diagonal elements are typically 1 (or set to the maximum value, indicating that the category is completely similar to itself). The nth element in the matrix... i Line number j The element of the column represents the first element. i The category and the first jThe visual association strength of each category in the feature space constructed by the current visual model constitutes the visual concept relationship matrix. Obtain the names or more descriptive text definitions of all target categories (e.g., "Type A valve: a double-flange cast steel valve for high-pressure pipelines"), which carry the semantic information of the categories. Input all the texts of these categories simultaneously or in pairs into the target multimodal large model. Utilize the powerful semantic understanding ability of the large model to obtain the strength of the relationship between them. Specific methods may include: (1) Similarity based on text embedding: Encode the text of each category separately with the text encoder of the large model to obtain the category text feature vector, and then calculate the cosine similarity between these vectors pairwise. This reflects the semantic similarity of the categories understood by the large model based on language modeling; (2) Association based on knowledge reasoning: If the large model has knowledge reasoning ability or access to a knowledge graph, it can calculate more complex semantic associations, such as kinship (whether they belong to the same parent class), component sharing relationship, functional complementarity relationship, etc., and quantify these relationships into a strength score. Similar to the visual matrix, fill another matrix with the semantic relationship strength between all the category pairs calculated, in the same category order. This semantic concept relation matrix reflects the prior association strength between various categories of concepts in the domain knowledge system (cognitive graph) contained in the target multimodal large model, which constitutes the semantic concept relation matrix. Calculate the difference between the visual concept relation matrix and the semantic concept relation matrix. Commonly used measurement methods include: (1) Mean squared error: directly calculate the average of the squares of the differences between corresponding elements of the two matrices, which forces each element of the visual relation matrix to move closer to the corresponding element of the semantic relation matrix; (2) KL divergence: if each row (or each row after softmax) is regarded as a distribution of the relationship between a category and all other categories, the KL divergence between these two distributions can be calculated, which focuses more on the distribution shape of the matching relationship; (3) Cosine similarity loss: calculate the cosine distance between the vectorized two matrices. Use this difference measure directly as the third loss. During training, the parameters of the initial visual model are optimized through backpropagation, with the aim of minimizing this difference.

[0102] In some embodiments, the step of fusing the probability distributions of the first category and the second category using a preset evidence theory to obtain the fusion confidence score of each candidate category specifically includes:

[0103] The first category probability distribution is transformed into a first basic confidence assignment to characterize the degree of support for each candidate category, and the second category probability distribution is transformed into a second basic confidence assignment to characterize the degree of support for each candidate category.

[0104] Calculate the conflict measure between the first basic confidence assignment and the second basic confidence assignment;

[0105] If the conflict metric is lower than a preset threshold, then the first basic confidence assignment and the second basic confidence assignment are combined according to the preset evidence theory combination rules to obtain a joint basic confidence assignment.

[0106] Based on the joint basic confidence assignment, the confidence function value and likelihood function value of each candidate category are calculated, and the fusion confidence of each candidate category is obtained by weighted averaging of the confidence function value and the likelihood function value.

[0107] The first category probability distribution is the probability vector output by the softmax layer after the target visual model performs forward propagation on the input image. For example, [0.7, 0.2, 0.1] indicates that the model believes the image belongs to categories A, B, and C with probabilities of 70%, 20%, and 10%, respectively. The second category probability distribution is the probability vector normalized by the target multimodal large model after semantic similarity matching between the input image and the text descriptions of all candidate categories. It represents the confidence level based on semantic knowledge matching. The core of evidence theory (such as Dempster-Shafer theory) is the basic confidence assignment, transforming each probability distribution into a basic confidence assignment. A simplified and commonly used transformation method is to assign confidence to each candidate category based on an adjustment factor, and allocate the remaining confidence to the entire set (i.e., the proposition "not knowing which specific category it belongs to"), this part is called uncertainty confidence. This operation yields the first and second basic confidence assignments. Conflict in evidence theory refers to the degree to which two sources of evidence completely contradict each other in their support for the same proposition. The conflict measure can be the conflict coefficient of the two confidence assignments before combination. A high conflict coefficient indicates a significant disagreement between the two sources of evidence regarding which categories are impossible (e.g., one strongly supports A, the other strongly supports B). The calculated conflict metric is compared to a preset threshold. If the conflict metric is below the threshold, it indicates that while the two models may differ in details, there is no fundamental, irreconcilable contradiction, and deep fusion can proceed to the next step. If the conflict metric is above or equal to the threshold, it indicates that the two models have made seriously conflicting decisions. In this case, the standard combination rule is not executed; instead, a safety mechanism is triggered. The safety mechanism can be: selecting the result of the single model with higher confidence, activating a backup model, or directly outputting uncertainty and leaving it to human judgment. This prevents erroneous decisions caused by forced fusion when there is serious conflict in the evidence. Application of the combination rule: When the conflict is low, a preset evidence theory combination rule (most commonly Dempster's combination rule) is used to combine the first and second basic confidence assignments. This rule uses a mathematical formula to calculate the confidence that the two sources of evidence jointly support a proposition (a single category or a set of categories) and redistributes the confidence caused by the conflict according to the preset formula. The result is a new joint basic confidence assignment that integrates all the evidence. The joint basic confidence assignment is more comprehensive and accurate than any single confidence assignment because it integrates information from two independent perspectives. Based on the joint basic confidence assignment, a confidence function and a likelihood function are calculated for each candidate category. The confidence function is equal to the sum of all confidence levels supporting the candidate category or a subset thereof. It represents the minimum confidence level where evidence clearly supports the category and is the lower bound of the confidence level. The likelihood function is equal to the sum of all confidence levels that do not conflict with the candidate category. It represents the maximum possible confidence level where there is no evidence against the category and is the upper bound of the confidence level.The fusion confidence score for each candidate category is obtained by taking a weighted average of the trust function value and the likelihood function value.

[0108] The above describes a visual recognition method that integrates a large model and a visual model in the embodiments of this application. The computer system in the embodiments of this application will be described in detail below in conjunction with the above-described visual recognition method that integrates a large model and a visual model.

[0109] Please see Figure 3 This is a schematic diagram of an exemplary hardware structure of a computer system in an embodiment of this application.

[0110] In some embodiments, the computer system 300 includes a computer device, which may be a terminal device. The computer device includes a processor 301, a memory 302, a sensor module 303, a communication module 304, an input device 305, and an output device 306 connected via a system bus. The processor 301 of the computer device provides computing and control capabilities. The memory 302 of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database is used to store data.

[0111] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0112] In some embodiments of this application, a computer-readable storage medium is provided, including instructions that, when executed on a computer system 300, cause the computer system 300 to perform a visual recognition method that fuses a large model and a visual model according to an embodiment of this application.

[0113] In some embodiments of this application, a computer program product is also provided, which, when run on a computer system 300, causes the computer system 300 to execute a visual recognition method that integrates a large model and a visual model according to an embodiment of this application.

[0114] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0115] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0116] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for visual recognition that integrates large-scale models and visual models, characterized in that, include: Textual information from the target domain is collected to form a domain knowledge set, which includes structured knowledge text and unstructured descriptive text. The domain knowledge set and the corresponding domain image are used as training pairs to perform comparative learning on a pre-trained visual-language multimodal large model, and an elastic weight consolidation strategy is adopted simultaneously to obtain the target multimodal large model. A predetermined number of initial sample images for the target recognition category are obtained. The initial sample images are input into the target multimodal large model to generate a fine-grained structured text description corresponding to each initial sample image, resulting in an initial sample-description pair. The fine-grained structured text description includes object category, component composition, attribute state, and scene context. Based on the initial sample-description pair, a synthetic image-combined description pair is generated through semantic decoupling and recombination. An augmented dataset is constructed based on the initial sample-description pair and the synthetic image-combined description pair. The initial visual model is trained using the augmented dataset by minimizing a collaborative loss function to obtain a target visual model. The collaborative loss function includes a first loss, a second loss, and a third loss. The first loss is the standard classification loss of the initial visual model calculated based on the category labels of the initial sample images in the augmented dataset. The second loss is the distance between the visual features extracted by the initial visual model from the initial sample images in the augmented dataset and the text features extracted by the target multimodal large model from the fine-grained structured text description corresponding to the augmented dataset. The third loss is the difference between the relative relationships between categories in the feature space of the initial visual model and the domain cognitive graph constructed by the target multimodal large model. In response to receiving a new input image, the system acquires in parallel the first category probability distribution output by the target visual model and the second category probability distribution output by the target multimodal large model based on semantic similarity matching. The system then fuses the first category probability distribution and the second category probability distribution using a preset evidence theory to obtain the fusion confidence of each candidate category. If the highest fusion confidence is higher than or equal to the current dynamic confidence threshold, the category corresponding to the highest fusion confidence is output as the final recognition result.

2. The visual recognition method based on the fusion of a large model and a visual model according to claim 1, characterized in that, The step of using the domain knowledge set and corresponding domain images as training pairs to perform comparative learning on the pre-trained visual-language multimodal large model, and simultaneously employing an elastic weight consolidation strategy to obtain the target multimodal large model, specifically includes: A batch of training pairs is selected. For the first training pair in the batch, the first optimization objective is to maximize the cosine similarity between image features and text features in the cross-modal embedding space. The second optimization objective is to minimize the cosine similarity between the image features of the first training pair and other text features in the batch, as well as the cosine similarity between the text features of the first training pair and other image features in the batch. The first training pair can be any training pair in the batch. The contrastive learning loss function is calculated based on the first optimization objective and the second optimization objective. The Fisher information matrix of each parameter of the visual-language multimodal large model is evaluated based on all training pairs, and the diagonal elements of the Fisher information matrix are taken to form the parameter importance vector. The regularization term is obtained based on the training difference between each element in the parameter importance vector and the corresponding parameter of the visual-language multimodal large model, and the contrastive learning loss function is added to the regularization term to construct a joint training loss function; With the goal of minimizing the joint training loss function, the vision-language multimodal large model is iteratively optimized, and the converged model is taken as the target multimodal large model.

3. The visual recognition method that integrates a large model and a visual model according to claim 2, characterized in that, The step of evaluating the Fisher information matrix of each parameter of the visual-language multimodal large model based on all training pairs, and taking the diagonal elements of the Fisher information matrix to form a parameter importance vector, specifically includes: For the second training pair among all training pairs, calculate the gradient of the contrastive learning loss function with respect to the model parameters, and record the gradient components of the parameters involved in the forward propagation paths of the image modality and the text modality in the second training pair respectively. The second training pair is any one of the training pairs among all training pairs. Iterate through all training pairs, and aggregate the gradient components of the target parameter that appear on all image modal forward paths and all text modal forward paths respectively. Calculate the mean of the squared gradients in the image modality and the text modality respectively, and use it as an importance estimate of the target parameter in the corresponding modality. The target parameter can be any parameter. The importance estimates of the target parameter in the image modality and the importance estimates in the text modality are weighted and summed according to preset fusion weights to obtain the multimodal fusion importance score of the target parameter; The multimodal fusion importance scores of all parameters are arranged in order to form an initial importance vector. The initial importance vector is then smoothed and normalized to obtain the parameter importance vector.

4. The visual recognition method based on the fusion of a large model and a visual model according to claim 1, characterized in that, The process of generating synthetic image-combined description pairs based on the initial sample-description pairs through semantic decoupling and recombination includes: The fine-grained structured text description in each initial sample-description pair is decoupled into an independent set of semantic units using the target multimodal large model. Each semantic unit in the set of semantic units corresponds to a component, attribute, or context label. Each semantic unit in the set of semantic units is text-encoded to obtain the corresponding unit text feature vector. The initial sample image is visually encoded, and the local visual feature vector corresponding to each semantic unit is extracted through a region attention mechanism. From the semantic unit sets decoupled from different initial sample-description pairs, one or more semantic units are selected respectively and combined into a new semantic unit combination. According to the semantic unit combination, the corresponding local visual feature vectors are selected from each source image, and spatially perceptual feature fusion is performed to generate synthetic image features. The synthetic image features are input into a pre-trained diffusion model or generative adversarial network to generate the synthetic image. The semantic units are combined and reconstructed into a combined text description that conforms to the grammatical structure to obtain the synthetic image-combined description pair.

5. The visual recognition method based on the fusion of a large model and a visual model according to claim 1, characterized in that, The second loss is calculated using the following steps: For the first target sample in the augmented dataset, the visual features of the initial sample image are extracted using the initial visual model, and the text features of the corresponding fine-grained structured text description are extracted using the target multimodal large model. The visual features and the text features are mapped to a common semantic contrast space, respectively. The first target sample is any initial sample-description pair in the augmented dataset. In the public semantic comparison space, the overall similarity between the mapped visual features and the mapped text features of the first target sample is calculated. The text description is decomposed into multiple semantic components, and the local text features of each semantic component are extracted. The local similarity between the mapped visual features and each of the local text features is calculated. Based on the image quality and the level of detail of the text description of the first target sample, the contribution weights of the overall similarity and the local similarity in the calculation are dynamically adjusted, and the overall similarity and the local similarity are weighted and summed based on the contribution weights to obtain the final semantic matching degree. An alignment loss term is constructed based on the final semantic matching degree. The value of the alignment loss term increases as the final semantic matching degree decreases. The second loss is obtained by averaging the alignment loss terms of all samples in the augmented dataset.

6. The method for visual recognition by fusing a large model and a visual model according to claim 1, characterized in that, The third loss is calculated using the following steps: For the target category, all initial sample images belonging to the target category in the augmented dataset are collected, the visual features of the initial sample images are extracted using the initial visual model, and the visual prototype features of the target category are obtained through aggregation operations. The visual prototype features of each of the multiple target categories are obtained, the similarity between each pair of the multiple visual prototype features is calculated, and a visual concept relationship matrix is ​​constructed. The elements in the visual concept relationship matrix represent the association strength between two categories in the visual feature space. Input the names or descriptive texts of multiple target categories into the target multimodal large model, extract the semantic relationship strength between the target categories through the target multimodal large model, and construct a semantic concept relationship matrix; The difference measure between the visual concept relation matrix and the semantic concept relation matrix is ​​calculated and used as the third loss.

7. The method for visual recognition by fusing a large model and a visual model according to claim 1, characterized in that, The process of fusing the probability distributions of the first and second categories using a pre-defined evidence theory to obtain the fusion confidence score for each candidate category specifically includes: The first category probability distribution is transformed into a first basic confidence assignment to characterize the degree of support for each candidate category, and the second category probability distribution is transformed into a second basic confidence assignment to characterize the degree of support for each candidate category. Calculate the conflict measure between the first basic confidence assignment and the second basic confidence assignment; If the conflict metric is lower than a preset threshold, then the first basic confidence assignment and the second basic confidence assignment are combined according to the preset evidence theory combination rules to obtain a joint basic confidence assignment. Based on the joint basic confidence assignment, the confidence function value and likelihood function value of each candidate category are calculated, and the fusion confidence of each candidate category is obtained by weighted averaging of the confidence function value and the likelihood function value.

8. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-7.

Images