Open vocabulary image recognition method, system and device and storage medium

Abstract

The invention provides an open vocabulary image recognition method, system and device and a storage medium. The method comprises the steps that a large language model is called based on a constructed structured semantic template to generate multi-vision semantic description; performing legality verification on the multi-visual semantic description by utilizing a knowledge graph to obtain a multi-description visual semantic prompt set; according to the multi-description visual semantic prompt set, combining a constructed image encoder and a text encoder, generating a self-adaptive weight, and generating a bidirectional loss function based on the self-adaptive weight; jointly optimizing the image encoder and the text encoder by using the bidirectional loss function; and executing open vocabulary image reasoning by utilizing the optimized image encoder and text encoder and combining the multi-description visual semantic prompt set to obtain a recognition result. By means of the method, the problem that traditional single prompt semantics are sparse is solved.

Description

An open vocabulary image recognition method, system, device, and storage medium Technical Field

[0001] This application relates to the field of computer vision, specifically to an open vocabulary image recognition method, system, device, and storage medium. Background Technology

[0002] Traditional image recognition methods rely on training on a fixed set of categories, only able to recognize categories known during the training phase. They cannot generalize to the constantly emerging new categories in the open world, suffering from a severe closed-vocabulary bottleneck. To overcome this limitation, open-vocabulary image recognition technology has rapidly developed in recent years. Its core idea is to train models using large-scale image and text data, aligning image embeddings with the text semantic space, thereby achieving zero-shot inference for unseen categories. These methods typically embed category names as text prompts and calculate their similarity to image features to determine the category.

[0003] However, existing mainstream solutions still face significant limitations. First, most methods rely on single, static template-based prompts, resulting in sparse semantic representations that fail to capture intra-class visual diversity. Second, while some research has attempted to introduce visual attributes to enrich textual descriptions, decoupling and modeling these attributes in isolation ignores the fundamental principle that objects in the real world are often defined by multiple co-occurring visual attributes. Furthermore, existing methods suffer from crude matching mechanisms, typically simply summing the matching degrees of images with multiple descriptions without dynamically adjusting the contribution weights of each description based on the actual content of the input image. They also lack robustness to real-world interference such as occlusion, perspective changes, or missing attributes. Therefore, a novel open vocabulary recognition framework is urgently needed that can both explicitly model the co-occurrence relationships of multiple visual attributes and adaptively fuse semantic signals from multiple descriptions. Summary of the Invention

[0004] To address the problem that existing technologies, in practical applications, suffer from single, static template-based prompts with sparse semantic expression, making it difficult to depict the visual diversity within a category, this application provides an open vocabulary image recognition method, system, device, and storage medium.

[0005] The first aspect of this application provides an open-vocabulary image recognition method, the method comprising: generating multi-visual semantic descriptions by calling a large language model based on a constructed structured semantic template; performing legality verification on the multi-visual semantic descriptions using a knowledge graph to obtain a multi-description visual semantic cue set; generating a bidirectional loss function by combining the multi-description visual semantic cue set with a constructed image encoder and a text encoder; jointly optimizing the image encoder and the text encoder using the bidirectional loss function; and performing open-vocabulary image inference using the optimized image encoder and text encoder in conjunction with the multi-description visual semantic cue set to obtain a recognition result.

[0006] In a possible implementation, the step of using a knowledge graph to perform legality verification on the multi-visual semantic descriptions to obtain a multi-description visual semantic prompt set includes: extracting structured triples for each of the multi-visual semantic descriptions; using the knowledge graph to perform attribute existence checks, value range rationality checks, and attribute combination consistency checks on each extracted structured triple; including multi-visual semantic descriptions that pass all checks into an initial prompt set; and feeding back visual semantic descriptions that fail the checks to the large language model to generate new multi-visual semantic descriptions, and repeatedly performing legality verification on the newly generated multi-visual semantic descriptions until a preset number of multi-visual semantic descriptions that pass the verification are obtained, thus constituting the multi-description visual semantic prompt set.

[0007] In a possible implementation, generating a bidirectional loss function based on the image encoder and text encoder constructed by combining the multi-description visual semantic cue set includes: constructing an image encoder and a text encoder; encoding the semantic descriptions in the multi-description visual semantic cue set using the text encoder to obtain text features; encoding the input image using the image encoder to obtain visual features; calculating similarity based on the text features and visual features; generating adaptive weights based on the similarity; using the adaptive weights combined with the text features to obtain a category semantic prototype; and generating a bidirectional loss function based on the category semantic prototype, a preset classification temperature coefficient, and the visual features.

[0008] In a possible implementation, the step of encoding the semantic descriptions in the multi-description visual semantic cue set to obtain text features according to the text encoder includes: encoding each subset of the multi-description visual semantic cue set and converting each subset into an encoding sequence of a preset length; inputting the encoding sequence into the text encoder and performing attention calculation and feature transformation sequentially through multiple encoder layers to obtain initial text features; and obtaining a normalized text feature vector based on the initial text features and the L2 norm of the initial text features.

[0009] In a possible implementation, encoding the input image to obtain visual features by the image encoder includes: extracting features from the input image through the backbone network of the image encoder to obtain initial visual features; and obtaining a normalized visual feature vector based on the initial visual features and their L2 norm.

[0010] In a possible implementation, the step of calculating similarity based on the text features and visual features, generating adaptive weights based on the similarity, and using the adaptive weights in combination with the text features to obtain a category semantic prototype includes: calculating cosine similarity based on the visual features and all text features under the corresponding category; normalizing the cosine similarity using a softmax function to generate dynamic weights, wherein the sharpness of the distribution of the dynamic weights is adjusted by a preset or learnable temperature coefficient; and performing a weighted summation on all text features under the corresponding category based on the dynamic weights to obtain a dynamic category semantic prototype dependent on the current input image, and normalizing the category semantic prototype.

[0011] In a possible implementation, generating a bidirectional loss function based on the category semantic prototype, a preset classification temperature coefficient, and visual features includes: calculating the image-to-text loss based on the category semantic prototype and the preset classification temperature coefficient; calculating the text-to-image loss based on the category semantic prototype and the visual features, wherein the text-to-image loss is obtained by calculating the similarity between visual features and text features in the same category and the difference in similarity between visual features and text features in different categories; and performing a weighted summation of the image-to-text loss and the text-to-image loss to obtain the bidirectional loss function.

[0012] A second aspect of this application provides an open vocabulary image recognition system, comprising: a calling module for generating multi-visual semantic descriptions by calling a large language model based on a constructed structured semantic template; a verification module for performing legality verification on the multi-visual semantic descriptions using a knowledge graph to obtain a multi-description visual semantic cue set; a loss function calculation module for generating a bidirectional loss function based on the multi-description visual semantic cue set combined with a constructed image encoder and a text encoder; an optimization module for jointly optimizing the image encoder and the text encoder using the bidirectional loss function; and a recognition module for performing open vocabulary image inference using the optimized image encoder and text encoder, combined with the multi-description visual semantic cue set, to obtain a recognition result.

[0013] A third aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described above.

[0014] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0015] As can be seen from the above technical solutions, this application defines a structured semantic template for visual attributes, ensuring that the generated descriptions only contain observable visual features and avoiding non-visual interference. It then calls a large language model to generate natural language descriptions with multiple co-occurring attributes, rather than using a single category name or isolated attributes. This multi-attribute combination description approach approximates the fine-grained perception mode of human objects, solving the problem of semantic sparsity in traditional single-prompt descriptions. The legality of the descriptions is verified through a knowledge graph, eliminating descriptions with contradictory attributes or that do not conform to real-world rules, ensuring that each description has visual feasibility and category specificity. This provides high-quality semantic priors for image-text matching, reducing cross-category confusion from the source. Furthermore, the unified image encoder and text encoder constructed in this application map visual features and text features to the same semantic space, ensuring the effectiveness of similarity calculation. Simultaneously, an adaptive aggregation mechanism dynamically allocates weights based on the semantic similarity between the image and each description, ensuring that descriptions highly matched to the image receive the final score. This effectively highlights discriminative information, suppresses noise interference from irrelevant descriptions, and further reduces the probability of misjudgment. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 is a flowchart illustrating an open vocabulary image recognition method according to an embodiment of this application.

[0018] Figure 2 is a schematic diagram of the knowledge-guided multi-descriptive visual semantic prompt generation process in an embodiment of this application.

[0019] Figure 3 is a schematic diagram of the image category determination process for multi-descriptive visual semantic prompts in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Based on this, this application provides an implementation method for open vocabulary image recognition, as shown in Figure 1. The method includes: S101, generating multi-visual semantic descriptions by calling a large language model based on a constructed structured semantic template; S102, performing legality verification on the multi-visual semantic descriptions using a knowledge graph to obtain a multi-description visual semantic prompt set; S103, generating a bidirectional loss function by combining the multi-description visual semantic prompt set with a constructed image encoder and text encoder; S104, jointly optimizing the image encoder and text encoder using the bidirectional loss function; S105, performing open vocabulary image reasoning using the optimized image encoder and text encoder, combined with the multi-description visual semantic prompt set, to obtain a recognition result.

[0022] It should be noted that the constructed structured semantic template provides a constraint framework for generating multi-descriptive visual semantic prompts for large language models, ensuring that the generated visual semantic descriptions focus on observable visual features and avoid introducing non-visual interference information. For example, the constructed structured semantic template must simultaneously satisfy two principles: full coverage of visual elements and strong exclusion of irrelevant content. On the one hand, it explicitly requires that the description include visual dimensions such as shape, key components, surface texture, color changes, and structural layout to ensure fine-grained semantic characterization. On the other hand, it strictly excludes non-visual concepts, abstract cultural references, and unrealistic variations to avoid a disconnect between textual semantics and visual representation. Furthermore, the structured semantic template of this application can flexibly adjust the priority of visual elements according to the target scenario. For example, in the scenario of industrial parts recognition, the description weight of key components and surface texture is strengthened; in the scenario of everyday consumer goods recognition, the focus is on the characterization of color changes and shapes. The structured semantic template can also introduce scenario-based guiding statements, such as describing visual features in conjunction with common usage environments of the category, to further improve the scenario adaptability of the description; this application does not impose any restrictions on this.

[0023] The process of generating a multi-description visual semantic prompt set based on structured knowledge graph verification in this application is shown in Figure 2. First, the category name is used as input and processed through the multi-description visual semantic prompt generation template. The output of this module is sent to the large language model, which generates multi-visual semantic descriptions. Subsequently, the multi-visual semantic descriptions enter the knowledge graph verification and judgment stage: if the multi-visual semantic descriptions pass the knowledge graph verification, they are output to the multi-description visual semantic prompt set on the right, which contains multiple entries such as description 1, description 2, ... description N; if the multi-visual semantic descriptions fail the knowledge graph verification, they are fed back to the large language model, which regenerates the multi-visual semantic descriptions and then enters the knowledge graph verification stage again until the multi-visual semantic descriptions pass the verification.

[0024] Each multi-descriptive visual semantic cue in this application is a complete natural language sentence that integrates multiple visual attributes that often co-occur in real images, explicitly modeling the contextual associations and combination rules between attributes such as color, shape, key components, pose, texture, and typical environment.

[0025] It should be noted that the multi-visual semantic descriptions generated by calling a large language model based on the constructed structured semantic template possess both diversity and standardization. Diversity is reflected in covering different visual variations of the same category, while standardization is reflected in each description starting with the category name and conforming to natural language expression habits. For example, when generating descriptions for the apple category, the large language model generates multiple descriptions based on the structured semantic template, such as "apple, red skin, round, with small spots on the surface, with a green stem," and "apple, green skin, oval, white flesh and brown core visible after slicing." This covers differences in color and state while adhering to the principle of prioritizing visual features. Optional large language models include, but are not limited to, Tongyi Qianwen, Zhipu Qingyan, DeepSeek, GPT-4, Claude, and Llama. In practical applications, an appropriate model can be selected based on the model's semantic understanding ability, generation efficiency, and deployment cost; this application does not impose any restrictions.

[0026] It should be noted that the length of the generated multi-visual semantic description needs to be controlled within the preset length to ensure that each multi-visual semantic description can fully contain the key visual features without affecting the subsequent encoding efficiency due to redundant information. Excessively long multi-visual semantic descriptions will introduce irrelevant visual details, while descriptions that are too short will not be able to fully depict the co-occurrence relationship of attributes.

[0027] This application defines a structured semantic template for visual attributes, ensuring that the generated descriptions contain only observable visual features and avoiding non-visual interference. It then uses a large language model to generate natural language descriptions with multiple co-occurring attributes, rather than using a single category name or isolated attributes. This multi-attribute combination approach approximates the fine-grained perception of objects by humans, solving the problem of semantic sparsity in traditional single-prompt descriptions. A knowledge graph is used to verify the legality of the descriptions, eliminating descriptions with contradictory attributes or that do not conform to real-world rules, ensuring that each description has visual feasibility and category specificity. This provides high-quality semantic priors for image-text matching, reducing cross-category confusion from the source. Furthermore, this application constructs a unified image encoder and text encoder, mapping visual and text features to the same semantic space, ensuring the effectiveness of similarity calculations. An adaptive aggregation mechanism dynamically allocates weights based on the semantic similarity between the image and each description, ensuring that descriptions highly matched to the image receive the highest score. This effectively highlights discriminative information, suppresses noise interference from irrelevant descriptions, and further reduces the probability of misjudgment. Finally, this application jointly optimizes the encoder through a bidirectional loss function, taking into account both image-to-text matching errors and text-to-image matching errors, thereby making the semantic alignment of image and text features more complete, avoiding modal shift caused by unidirectional optimization, and providing a cross-modal feature foundation for accurate recognition.

[0028] In one embodiment of this application, the step of using a knowledge graph to perform legality verification on the multi-visual semantic descriptions to obtain a multi-description visual semantic prompt set includes: S201, extracting structured triples for each of the multi-visual semantic descriptions; S202, using a knowledge graph to perform attribute existence checks, value range rationality checks, and attribute combination consistency checks on each extracted structured triple; S203, incorporating the multi-visual semantic descriptions that pass all checks into the initial prompt set; S204, feeding back the visual semantic descriptions that fail the checks to the large language model to generate new multi-visual semantic descriptions, and repeatedly performing legality verification on the newly generated multi-visual semantic descriptions until a preset number of multi-visual semantic descriptions that pass the verification are obtained, constituting the multi-description visual semantic prompt set.

[0029] This application first uses an information extraction tool to process each generated multi-visual semantic description. For example, it extracts structured triples in the form of <subject, attribute, value>. For the description of an apple, red skin, round shape, with a green stem, it extracts three triples: <apple, skin color, red>, <apple, shape, round>, and <apple, stem color, green>.

[0030] It should be noted that the accuracy of triple extraction directly affects the subsequent verification results. Therefore, the extraction tool needs to be preprocessed and optimized, for example, by adjusting the part-of-speech tagging rules through a domain dictionary to ensure the accurate division of subject, attribute, and value.

[0031] In this embodiment, a pre-defined knowledge graph is used to perform three-layer checks on the extracted triples, including: attribute existence check, value range reasonableness check, and attribute combination consistency check. The attribute existence check determines whether the subject naturally possesses the attribute by querying the knowledge graph. For example, the attributes of apple in the knowledge graph include peel color, shape, and stem color; therefore, <apple, peel color, red> passes the attribute existence check. If <apple, engine type, gasoline> is extracted, it fails the attribute existence check because apples do not possess the engine type attribute. The value range reasonableness check determines whether the value belongs to a reasonable range of values ​​for the attribute by querying the knowledge graph. For example, reasonable values ​​for apple peel color include red, green, and yellow; therefore, <apple, peel color, blue> fails the value range reasonableness check. The attribute combination consistency check determines whether there are logical contradictions in multiple attribute combinations of the same subject by calling the rule base in the knowledge graph. For example, if both <apple, state, complete> and <apple, state, slice> are extracted at the same time, the attribute combination consistency check will fail because complete and slice are antonyms.

[0032] It should be noted that the choice of knowledge graph needs to be determined according to the recognition scenario. General scenarios use general graphs such as Wikidata, while professional scenarios such as medical image recognition and industrial parts recognition use domain-specific graphs, such as UMLS in the medical field. This application does not impose any restrictions on this.

[0033] In this embodiment, the visual semantic descriptions corresponding to the triples that pass the three-layer validation are included in the initial prompt set. Descriptions that fail validation are fed back to the large language model along with the reasons for failure, guiding the model to regenerate new descriptions. For example, if the description "apple, blue skin" fails the value range validity check because <apple, skin color, blue>, the feedback is that apple skin color does not have blue, and a description conforming to real-world rules should be regenerated. The model then generates "apple, yellow skin". It should be noted that the iterative process needs to continue until a predetermined number of validated descriptions are generated for the target category, constituting the final multi-description visual semantic prompt set.

[0034] It should be noted that the multi-description visual semantic cue set mentioned in this application refers to a set of natural language descriptions containing N co-occurring visual attributes that have been verified by a knowledge graph.

[0035] In one embodiment of this application, the step of generating a bidirectional loss function based on the image encoder and text encoder constructed by combining the multi-description visual semantic cue set includes: S301, constructing an image encoder and a text encoder, encoding the semantic descriptions in the multi-description visual semantic cue set using the text encoder to obtain text features, and encoding the input image using the image encoder to obtain visual features; S302, calculating similarity based on the text features and visual features, generating adaptive weights based on the similarity, and using the adaptive weights combined with the text features to obtain a category semantic prototype; S303, generating a bidirectional loss function based on the category semantic prototype, a preset classification temperature coefficient, and visual features.

[0036] In this embodiment, the core of the encoder construction module is to construct an image encoder and a text encoder to achieve alignment of visual features and text features in the same semantic space. It should be noted that the design of the image encoder and the text encoder meets the two major requirements of consistent feature dimensions and normalization processing to ensure the effectiveness of subsequent similarity calculation.

[0037] In one embodiment of this application, the step of encoding the semantic descriptions in the multi-description visual semantic cue set according to the text encoder to obtain text features includes: S401, encoding each subset of the multi-description visual semantic cue set and converting each subset into an encoding sequence of a preset length; S402, inputting the encoding sequence into the text encoder and performing attention calculation and feature transformation sequentially through multiple encoder layers to obtain initial text features; S403, obtaining a normalized text feature vector based on the initial text features and the L2 norm of the initial text features.

[0038] It should be noted that in this embodiment, the text encoder adopts a Transformer structure. Its input is a single description from a multi-description visual semantic cue set, and its output is a text feature vector normalized by the L2 norm. The specific encoding process is as follows: the input description is tokenized, converting natural language into a token sequence that the model can recognize; initial text features are extracted through the Transformer encoder layer. The initial text features are normalized using the L2 norm to obtain the final text features. The calculation formula is: in, Representing initial text features The L2 norm is calculated as follows: , for The The values ​​that can be taken in each dimension.

[0039] For example, to describe an apple—red skin, round shape—the text encoder first converts it into a token sequence, then uses multi-head attention to capture the association between the red skin and the round shape, ultimately outputting a dimension of... Normalized text features.

[0040] It should be noted that the number of Transformer layers and attention heads in the text encoder are adjusted according to the complexity of the semantic description. The more attributes the semantic description contains, the more layers need to be added to fully capture the contextual relationships between attributes.

[0041] In one embodiment of this application, the step of encoding the input image to obtain visual features by the image encoder includes: S501, extracting features from the input image through the backbone network of the image encoder to obtain initial visual features; S502, obtaining a normalized visual feature vector based on the initial visual features and the L2 norm of the initial visual features.

[0042] It should be noted that in this embodiment, the image encoder uses Vision Transformer or ResNet as the backbone network. Its input is the original image, and its output is a visual feature vector normalized to the L2 norm. The specific encoding process is as follows: the input image undergoes preprocessing, such as resizing and normalization; the initial visual features of the image are extracted through the backbone network. The initial visual features are normalized using the L2 norm to obtain the final visual features. The calculation formula is: ,in, Representing initial visual features The L2 norm is calculated as follows: , for The The values ​​that can be taken in each dimension.

[0043] It should be noted that the advantage of using Vision Transformer as the backbone network lies in its ability to capture global image features, making it suitable for characterizing the overall shape and attribute combination of objects. ResNet is suitable for processing small-sized images or scenarios with limited computing resources, and this application does not impose any restrictions on this.

[0044] In one embodiment of this application, the step of calculating similarity based on the text features and visual features, generating adaptive weights based on the similarity, and using the adaptive weights in combination with the text features to obtain a category semantic prototype includes: S601, calculating cosine similarity based on the visual features and all text features under the corresponding category; S602, normalizing the cosine similarity using a softmax function to generate dynamic weights, wherein the sharpness of the distribution of the dynamic weights is adjusted by a preset or learnable temperature coefficient; S603, performing weighted summation on all text features under the corresponding category based on the dynamic weights to obtain a dynamic category semantic prototype dependent on the current input image, and normalizing the category semantic prototype.

[0045] In this embodiment, the cosine similarity between visual features and all text features under the corresponding category is first calculated to measure the semantic correlation between them. For each category... The Text features corresponding to each text prompt and the visual features of the input image. cosine similarity The calculation formula is: ,in, express and The dot product, i.e. , for The Each dimension can take values. for The Each dimension can take values; , They are respectively and The L2 norm of is normalized, therefore , In actual calculations, it is simplified to .

[0046] It should be noted that the range of values ​​for cosine similarity is [range missing]. The closer the value is to 1, the stronger the semantic association between visual features and text features; the closer the value is to -1, the weaker the semantic association between visual features and text features.

[0047] In this embodiment of the application, based on similarity vectors Where N is the number of descriptions for category c, and the dynamic weights are generated by normalization using the softmax function. The calculation formula is: ,in, , This is a preset temperature coefficient used to adjust the sharpness of the weight distribution. The smaller the value, the more concentrated the weight distribution. The larger the value, the more evenly the weight distribution.

[0048] For example, if the input image is a whole red apple, then the similarity to descriptions of apples, red skin, and round shape is... The weight of this description is relatively high after softmax calculation. The weighting will be significantly higher than that associated with apples, green slices, and fruits with cores. It should be noted that the temperature coefficient... The settings need to be adjusted based on the characteristics of the dataset. If the dataset contains images with rich visual variations, a smaller setting is preferable. To highlight the key descriptions, if the image has significant occlusion, set a larger value. This is to avoid matching errors caused by a single invalid description.

[0049] In this embodiment, based on dynamic weights , for category All text features Perform a weighted summation to obtain the category. Dynamic semantic prototype The calculation formula is: Subsequently, on Perform L2 norm normalization to ensure it matches the scale of visual features. The normalization formula is as follows: .

[0050] It should be noted that, The core characteristic is image dependency, within the same category. When faced with different images, different semantic prototypes are generated depending on the activated descriptive subsets. For example, when faced with a whole red apple and a sliced ​​green apple, the categories... of It will favor text features related to red, complete and green, and slices respectively, thereby improving the accuracy of matching.

[0051] In one embodiment of this application, generating a bidirectional loss function based on the category semantic prototype, a preset classification temperature coefficient, and visual features includes: S701, calculating the image-to-text loss based on the category semantic prototype and the preset classification temperature coefficient; S702, calculating the text-to-image loss based on the category semantic prototype and the visual features, wherein the text-to-image loss is obtained by calculating the similarity between visual features and text features in the same category and the difference in similarity between visual features and text features in different categories; S703, performing a weighted summation of the image-to-text loss and the text-to-image loss to obtain the bidirectional loss function.

[0052] In this embodiment, the goal of the joint optimization module is to simultaneously optimize the image encoder and text encoder by calculating a bidirectional loss function, thereby achieving semantic alignment between visual features and text features. It should be noted that the design of the loss function must consider both image-to-text and text-to-image matching errors to avoid modal shift caused by unidirectional optimization.

[0053] In this embodiment, the bidirectional loss function is a loss function that converts images to text. Text-to-image loss The weighted sum yields the total loss. The calculation formula is: ,in, As a scale factor, in the embodiments of this application, Setting it to 1 ensures that the weights of the two types of losses are consistent; this application does not impose any restrictions on this. Specifically, Multi-class cross-entropy loss is used to measure the matching error between image features and the true category semantic prototypes. For the input image... First, calculate image features. semantic prototypes of all candidate categories Match score Subsequently, a preset classification temperature coefficient is introduced. ,calculate ,in, Image features Compared to the real category semantic prototype Match score, for With category semantic prototype The matching score. It should be noted that... Its function is to adjust the distribution of classification probabilities. The smaller the value, the more concentrated the probability of the correct class, and the larger the gradient during model training. The larger the value, the flatter the probability distribution, which helps avoid model overfitting.

[0054] To measure the matching error between text features and true category visual features, a contrastive learning approach is used. For images within a batch, a category is defined. The image index set within a batch is The formula for converting text to an image is: ,in, index within a batch Image features With index Image features ; for With index Image features Similarity; This is the temperature coefficient.

[0055] It should be noted that, The core idea is to bring image features of the same category closer together and widen the distance between image features of different categories, ensuring that the visual features corresponding to text features have clustering properties in the semantic space.

[0056] The joint optimization in this application adopts a hierarchical fine-tuning strategy. First, the parameters of the first few layers of the image encoder and text encoder are fixed, and only the parameters of the last few layers are fine-tuned. During training, the Adam optimizer is used to update the trainable parameters of the image encoder and text encoder simultaneously.

[0057] For example, for the ViT-B / 16 image encoder, the parameters of the first 6 Transformer layers can be fixed, and the parameters of the last 6 layers can be fine-tuned; for the CLIP text encoder, the parameters of the first 4 Transformer layers can be fixed, and the parameters of the last 4 layers can be fine-tuned. It should be noted that the choice of the number of fixed layers depends on the size of the pre-trained model and the complexity of the dataset. The larger the pre-trained model and the smaller the dataset, the more layers can be fixed to avoid overfitting; conversely, the number of fixed layers can be reduced to improve the model's adaptability.

[0058] It should be noted that this application also includes category prediction in open vocabulary scenarios. By constructing multiple natural language descriptions focusing on observable visual attributes for each category, the semantic richness and discriminative ability of the text are significantly improved. At the same time, an adaptive prompt aggregation mechanism based on semantic consistency is introduced to adaptively strengthen highly matching descriptions and suppress irrelevant or conflicting descriptions according to the image content, effectively enhancing the robustness and noise resistance of the matching process. In one embodiment of this application, the category prediction process is shown in Figure 3. The process includes two types of inputs: descriptions 1 to N of category c and corresponding image examples. Then, descriptions 1 to N of category c are input into a text encoder, which generates text embeddings corresponding to each description after processing. Specifically, text embeddings 1 to N are used to distinguish each embedding with a different pattern style. At the same time, the corresponding image examples are input into an image encoder, which generates corresponding image embeddings represented by a specific pattern style after processing. Then, the above-mentioned text embeddings 1 to N and the image embeddings are used together to perform adaptive aggregation processing using semantic consistency. Based on semantic association, all text embeddings and image embeddings are aggregated, and the aggregation result is output in the form of a combined pattern style. Finally, the aggregation result is used to calculate similarity, and the category score distribution is output after calculation, which is displayed in the form of a bar chart to show the scores of different categories.

[0059] This step not only significantly surpasses existing technologies in zero-shot or few-shot image classification tasks, but the trained model can also serve as a general visual semantic foundation model, directly transferable to downstream tasks such as open-vocabulary object detection, semantic segmentation, and image-text retrieval, significantly improving the model's generalization ability to unseen categories and its practical deployment value.

[0060] It should be noted that candidate categories can include categories that did not appear during the training phase. For new categories that did not appear, it is sufficient to generate a set of multi-description visual semantic cues that meet the requirements, without retraining the encoder.

[0061] The input image is processed by an optimized image encoder. Encode to obtain normalized visual features ; for each candidate category, an optimized text encoder is used. Encode the multi-description hint set to obtain the text feature set. For each candidate category ,calculate Cosine similarity with all text features in this category Dynamic weights are generated using the softmax function. Subsequently, based on dynamic weights By fusing text features, categories can be obtained. Dynamic semantic prototype And perform L2 norm normalization, and finally calculate With each candidate category The semantic prototype of (c=1, …, C', where C' is the number of candidate categories) Match score The category with the highest matching score is determined as the predicted category of the input image.

[0062] For example, if the candidate category includes the Satsuma mandarin orange which was not seen during training, a set of multiple descriptive prompts is generated for Satsuma mandarin orange, which has an orange peel, is round, and has pits on its surface; and Satsuma mandarin orange has a green stem and has segmented flesh after peeling. The prediction and recognition are completed through the above process.

[0063] It should be noted that the number of candidate categories in the inference stage is unlimited and can be dynamically increased or decreased according to the actual application scenario. For example, in a retail scenario, the candidate categories can include hundreds of products, while in an industrial quality inspection scenario, the candidate categories can include dozens of parts.

[0064] Furthermore, this application provides an open vocabulary image recognition system, the system comprising: a calling module 801, used to call a large language model based on a constructed structured semantic template to generate multi-visual semantic descriptions; a verification module 802, used to perform legality verification on the multi-visual semantic descriptions using a knowledge graph to obtain a multi-description visual semantic prompt set; a loss function calculation module 803, used to generate a bidirectional loss function based on the multi-description visual semantic prompt set combined with a constructed image encoder and text encoder; an optimization module 804, used to jointly optimize the image encoder and text encoder using the bidirectional loss function; and a recognition module 805, used to perform open vocabulary image reasoning using the optimized image encoder and text encoder, combined with the multi-description visual semantic prompt set, to obtain a recognition result.

[0065] From a hardware perspective, to provide an embodiment of an electronic device for implementing all or part of the open vocabulary image recognition method, the electronic device specifically includes the following components: a processor, a memory, a communication interface, and a bus. The processor, memory, and communication interface communicate with each other via the bus. The communication interface is used to realize information transmission between servers, devices, distributed message middleware cluster devices, various databases, and user terminals. The electronic device can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the electronic device can be implemented with reference to the embodiments of the open vocabulary image recognition method and the open vocabulary image recognition system, the contents of which are incorporated herein by reference, and repeated details are not repeated.

[0066] Embodiments of the present invention also provide a computer-readable storage medium capable of implementing all steps of the open vocabulary image recognition method in the above embodiments, wherein the execution subject can be a server. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the address association method in the above embodiments.

[0067] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0069] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0070] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0071] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. An open vocabulary image recognition method, characterized in that, include: The constructed structured semantic template is used to call a large language model to generate multi-visual semantic descriptions; The validity of the multi-visual semantic descriptions is verified using a knowledge graph to obtain a multi-description visual semantic prompt set. Adaptive weights are generated by combining the constructed image encoder and text encoder with the multi-description visual semantic prompt set, and a bidirectional loss function is generated based on the adaptive weights. The bidirectional loss function is used to jointly optimize the image encoder and text encoder. Open-vocabulary image reasoning is performed using the optimized image encoder and text encoder in combination with the multi-description visual semantic prompt set to obtain the recognition result.

2. The open vocabulary image recognition method according to claim 1, characterized in that, The step of using a knowledge graph to perform legality verification on the multi-visual semantic descriptions to obtain a multi-description visual semantic prompt set includes: extracting structured triples for each multi-visual semantic description; using the knowledge graph to perform attribute existence checks, value range rationality checks, and attribute combination consistency checks on each extracted structured triple; including multi-visual semantic descriptions that pass all checks into the initial prompt set; and feeding back visual semantic descriptions that fail the checks to the large language model to generate new multi-visual semantic descriptions, and repeatedly performing legality verification on the newly generated multi-visual semantic descriptions until a preset number of multi-visual semantic descriptions that pass the verification are obtained, thus constituting the multi-description visual semantic prompt set.

3. The open vocabulary image recognition method according to claim 1, characterized in that, The step of generating adaptive weights based on the image encoder and text encoder constructed by combining the multi-description visual semantic cue set, and generating a bidirectional loss function based on the adaptive weights, includes: constructing an image encoder and a text encoder; encoding the semantic descriptions in the multi-description visual semantic cue set using the text encoder to obtain text features; encoding the input image using the image encoder to obtain visual features; calculating similarity based on the text features and visual features; generating adaptive weights based on the similarity; using the adaptive weights combined with the text features to obtain a category semantic prototype; and generating a bidirectional loss function based on the category semantic prototype, a preset classification temperature coefficient, and the visual features.

4. The open vocabulary image recognition method according to claim 3, characterized in that, The step of encoding the semantic descriptions in the multi-description visual semantic cue set to obtain text features according to the text encoder includes: encoding each subset of the multi-description visual semantic cue set and converting each subset into an encoding sequence of a preset length; inputting the encoding sequence into the text encoder and performing attention calculation and feature transformation sequentially through multiple encoder layers to obtain initial text features; and obtaining a normalized text feature vector based on the initial text features and the L2 norm of the initial text features.

5. The open vocabulary image recognition method according to claim 3, characterized in that, The step of encoding the input image to obtain visual features by the image encoder includes: extracting features from the input image through the backbone network of the image encoder to obtain initial visual features; and obtaining a normalized visual feature vector based on the initial visual features and their L2 norm.

6. The open vocabulary image recognition method according to claim 3, characterized in that, The step of calculating similarity based on the text features and visual features, generating adaptive weights based on the similarity, and using the adaptive weights in combination with the text features to obtain a category semantic prototype includes: calculating cosine similarity based on the visual features and all text features under the corresponding category; normalizing the cosine similarity using a softmax function to generate dynamic weights, wherein the sharpness of the distribution of the dynamic weights is adjusted by a preset or learnable temperature coefficient; and performing a weighted summation on all text features under the corresponding category based on the dynamic weights to obtain a dynamic category semantic prototype dependent on the current input image, and normalizing the category semantic prototype.

7. The open vocabulary image recognition method according to claim 3, characterized in that, The step of generating a bidirectional loss function based on the category semantic prototype, a preset classification temperature coefficient, and visual features includes: calculating the image-to-text loss based on the category semantic prototype and the preset classification temperature coefficient; calculating the text-to-image loss based on the category semantic prototype and the visual features, wherein the text-to-image loss is obtained by calculating the similarity between visual features and text features in the same category and the difference in similarity between visual features and text features in different categories; and performing a weighted summation of the image-to-text loss and the text-to-image loss to obtain the bidirectional loss function.

8. An open vocabulary image recognition system, characterized in that, The system includes: a calling module for calling a large language model based on a constructed structured semantic template to generate multi-visual semantic descriptions; a verification module for using a knowledge graph to verify the legality of the multi-visual semantic descriptions and obtain a multi-description visual semantic prompt set; a loss function calculation module for generating a bidirectional loss function based on the multi-description visual semantic prompt set and a constructed image encoder and text encoder; an optimization module for jointly optimizing the image encoder and text encoder using the bidirectional loss function; and a recognition module for performing open-vocabulary image reasoning using the optimized image encoder and text encoder, combined with the multi-description visual semantic prompt set, to obtain a recognition result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images