A cross-modal conditional controllable retrieval method and system based on hybrid expert system

Abstract

The present invention discloses a cross-modal conditional controllable retrieval method and system based on a hybrid expert system, which belongs to the field of image retrieval technology. Using an image training dataset with style labels and content labels, a hybrid expert system based on a gated network and a prompt word enhancement vector group are trained in conjunction with contrastive learning loss and prompt regularization loss. The sum of the fusion features output by the hybrid expert system and the corresponding prompt vectors in the prompt word enhancement vector group is used as the image embedding vector for similarity retrieval; in the training phase, the fusion features output by the hybrid expert system are always weighted image features. In the retrieval phase, the retrieval mode is first determined according to the query text input by the user, and image retrieval is performed under different retrieval modes. The present invention combines the advantages of text and image modalities, can dynamically adjust the content and style retrieval weights, and based on a new prompt learning strategy, enables the model to adaptively select task-related prompts and flexibly adapt to different retrieval task requirements.

Description

Technical Field

[0001] The present invention relates to the field of image retrieval technology, and in particular to a cross-modal conditional controllable retrieval method and system based on a hybrid expert system. Background Art

[0002] With the widespread use of multimodal data, effectively extracting and matching relevant information from different types of media (such as text and images) has become a major challenge in the field of information retrieval. Existing image retrieval methods mostly focus on a single modality (e.g., content-based image retrieval or style-based image retrieval). However, in many practical applications, users need to consider both content and style information when searching for works of art. For example, when searching for works of art, users may be interested in both the specific content and the artistic style. Therefore, how to effectively combine content and style in image retrieval to meet users' multi-dimensional needs is a current research hotspot. Existing multimodal retrieval methods often rely on a single retrieval model or fixed input method, and are unable to flexibly adapt to the varying demands of the task. Therefore, how to design a retrieval method that can handle multimodal information and flexibly adapt to user needs has become a difficult problem that needs to be solved in this field. Summary of the Invention

[0003] In order to address the defects in the existing technology, the present invention proposes a cross-modal conditional controllable retrieval method and system based on a hybrid expert system, which combines the advantages of text and image modalities, can dynamically adjust the content and style retrieval weights, so that the retrieval can flexibly adapt to different task requirements. It is suitable for joint retrieval between image and text data, including content retrieval, style retrieval and content-style retrieval, and has wide applications in fields such as artistic style, design materials and cross-modal content retrieval.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] In a first aspect, a cross-modal conditional controllable retrieval method based on a hybrid expert system for image retrieval includes:

[0006] Obtain an image training dataset with style and content labels;

[0007] Constructing sample pairs based on the image training dataset, jointly training a hybrid expert system based on a gated network and a cue word enhanced vector group using contrastive learning loss and cue regularization loss, and adding the fusion features output by the hybrid expert system and the corresponding cue vectors in the cue word enhanced vector group as the image embedding vector for similarity retrieval;

[0008] During the training phase, the fusion features output by the hybrid expert system are weighted image features;

[0009] In the retrieval stage, the retrieval mode is first determined based on the query text entered by the user. In the content retrieval mode and style retrieval mode, the fusion features output by the hybrid expert system are the image features output by the corresponding expert model. In the content-style joint retrieval mode, the fusion features output by the hybrid expert system are weighted image features.

[0010] The hybrid expert system includes a pre-trained content expert model and a pre-trained style expert model for generating content image features and style image features respectively; the weighted image features refer to the weighted results of the fusion weight output by the gated network and the content image features and style image features.

[0011] Furthermore, the prompt word enhanced vector group includes a content prompt vector, a style prompt vector, and a content-style joint prompt vector. The three prompt vectors correspond to three retrieval modes respectively, and the initialization dimension of the prompt vector is the same as the fusion feature dimension.

[0012] Furthermore, the joint content-style cue vector is fixed and does not participate in training.

[0013] Furthermore, the first term of the hint regularization loss is the negative value of the L1 norm of the style hint vector and the content hint vector, and the second term is a regularization term representing the sum of the squares of the L2 norms of each dimension of the style hint vector and the content hint vector.

[0014] Furthermore, the gating network takes the text features of the query text as input and generates fusion weights based on probability distribution; the fusion weights include content weights, style weights and content-style joint weights.

[0015] Furthermore, the gating network is composed of an encoder based on a Transformer structure and a softmax function. The encoder takes the text features of the query text as input to generate encoding features, and the softmax function processes the encoding features into probability distribution.

[0016] Furthermore, the corresponding prompt vector in the prompt word enhanced vector group is determined according to the fusion weight output by the gated network, and the prompt vector of the retrieval mode corresponding to the maximum value of the fusion weight is taken.

[0017] Furthermore, during the training phase, the query text of each sample pair is randomly selected from content query text, style query text, or content-style joint query text. If the sample pair matches the query text, the sample pair label is a positive label, otherwise it is a negative label.

[0018] Furthermore, three databases are constructed in the retrieval phase:

[0019] A content database, applied to the content retrieval mode, stores the content image features of candidate images superimposed with content hint vectors;

[0020] The style database is used in the style retrieval mode, which stores the style image features of the candidate images after superimposing the style hint vector;

[0021] The content-style joint database is applied to the content-style joint retrieval mode, which stores the weighted image features of candidate images. The weighted fusion weights are generated through a gating network.

[0022] In a second aspect, the present invention proposes a cross-modal conditional controllable retrieval system based on a hybrid expert system, which is used to implement the above-mentioned cross-modal conditional controllable retrieval method based on a hybrid expert system.

[0023] The beneficial effects of the present invention are:

[0024] This paper proposes a hybrid expert system based on a gated network and a cue-word enhanced vector set. The hybrid expert system includes pre-trained content expert models and pre-trained style expert models for generating content image features and style image features, respectively. The gated network generates fusion weights for the image features generated by the different expert models and dynamically selects cue vectors based on the weights. The sum of the fused features and the corresponding cue vectors in the cue-word enhanced vector set serves as the image embedding vector for similarity retrieval. During the training phase, the fused features output by the hybrid expert system are always weighted image features. During the retrieval phase, the retrieval mode is determined based on the query text entered by the user, and image retrieval is performed under different retrieval modes.

[0025] This paper combines information from both text and images, overcoming the limitations of single-modal input. It applies contrastive learning to conditional retrieval tasks and introduces a hybrid expert system to enable collaboration between multiple expert models. Furthermore, it employs a novel prompt learning strategy that enables the model to adaptively select task-relevant prompts, thereby improving task focus.

[0026] The present invention not only supports content retrieval and style retrieval, but also can realize joint retrieval of content and style. It has strong controllability, and users can adjust the retrieval content and style preferences according to the input text; the present invention has excellent effects in multiple retrieval modes, especially in complex scenarios such as artwork retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of the cross-modal conditional controllable retrieval method based on a hybrid expert system proposed in the present invention.

[0028] Figure 2 This is the framework diagram of the training phase.

[0029] Figure 3 This is a simplified framework diagram of the retrieval stage (the prompt part is omitted).

[0030] Figure 4 It is a framework diagram of the prompt learning mechanism.

[0031] Figure 5 Schematic diagram of the application scenario of the present invention.

[0032] Figure 6 This is a schematic diagram of the training set data construction.

[0033] Figure 7 This is a comparison result diagram between the present invention and a typical retrieval model.

[0034] Figure 8 This is a schematic diagram of the perceptual evaluation results of different retrieval models by experts in the art field. DETAILED DESCRIPTION

[0035] The present invention will be further described and illustrated below in conjunction with specific embodiments. The embodiments are merely illustrative of the present disclosure and do not limit its scope. The technical features of the various embodiments of the present invention may be combined accordingly, provided that there is no conflict between them.

[0036] The accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0037] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to actual circumstances.

[0038] like Figure 1 As shown, the present invention proposes a cross-modal conditional controllable retrieval method based on a hybrid expert system for image retrieval, which mainly includes the following steps:

[0039] S01, obtain an image training dataset with style labels and content labels.

[0040] S02, construct sample pairs based on the image training dataset, and jointly use contrastive learning loss and prompt regularization loss to train a hybrid expert system based on a gated network and a prompt word enhanced vector group.

[0041] Here, the sum of the fusion features output by the hybrid expert system and the corresponding prompt vectors in the prompt word enhanced vector group is used as the image embedding vector for similarity retrieval; in the training stage, the fusion features output by the hybrid expert system are weighted image features.

[0042] S03, determining a retrieval mode according to the query text input by the user, and performing image retrieval under different retrieval modes.

[0043] Here, in the content retrieval mode and style retrieval mode, the fusion features output by the hybrid expert system are the image features output by the corresponding expert model. In the content-style joint retrieval mode, the fusion features output by the hybrid expert system are weighted image features.

[0044] In the present invention, the hybrid expert system includes a pre-trained content expert model and a pre-trained style expert model for generating content image features and style image features respectively; the weighted image features refer to the weighted results of the fusion weights output by the gated network and the content image features and style image features.

[0045] The training phase and inference phase are introduced below.

[0046] (1) Training phase

[0047] like Figure 2 As shown, the query text is generated through a text encoder to generate text features, while the content expert model and style expert model generate corresponding content features and style features for the image, respectively. Subsequently, the gating network dynamically calculates weights based on the text features, combines the content features and style features to generate fused features. A hint learning mechanism is used to enhance the model's focus on the task, resulting in an image embedding vector for similarity retrieval.

[0048] In a specific implementation of the present invention, the training phase is implemented as follows:

[0049] S11, build training set

[0050] Consider N labeled image samples , where each image Corresponding tags Defined as In this embodiment, the label satisfies:

[0051] Style Tags , representing 27 style category numbers, such as classicism, impressionism, abstractionism, minimalism, oil painting, Chinese style, sketching, watercolor, documentary photography, etc.

[0052] Content Tags , representing 80 object category numbers, such as buildings, vehicles (cars, airplanes, ships, bicycles), animals and plants (trees, flowers, cats, birds, marine life), natural phenomena (mountains, rivers, starry sky, lightning, volcanoes), furniture, books, backpacks, beaches, etc.

[0053] S12, each image The image is input into the expert model in the MOE system for processing, that is, the image is input into the content expert model and the style expert model respectively to extract the content features. and style characteristics .

[0054] The final output fusion features for:

[0055]

[0056] in, and It is dynamically generated by the gating network and generates different weights according to different query texts. The query texts are randomly selected from content query texts, style query texts, or content-style joint query texts.

[0057] For example, the content query text is "I want a picture with similar content"; the style query text is "I want a picture with a similar style"; and the content-style joint query text is "I want a picture with similar style and content".

[0058] S13, introduce the prompt word enhancement vector to the fusion feature of the final output of the MOE system , get the embedding vector for query :

[0059]

[0060] Here, the cue word enhancement vector is a learnable vector and is dynamically selected based on the output of the gating network. In order to avoid cue overfitting, a cue regularization loss is introduced, which will be explained below.

[0061] S14, based on this embedding vector , calculate the normalized cosine similarity of the image pair:

[0062]

[0063] Compute contrastive learning loss based on cosine similarity:

[0064]

[0065] in, represents contrast loss, N represents the number of training samples, Represents the temperature parameter, which is used to adjust contrast sensitivity; Respectively represent the similarity between training samples i and k, and between training samples i and j. The label indicating whether two training samples match is as follows:

[0066]

[0067] The above loss is used to push the positive samples to be more similar and the negative samples to be farther apart.

[0068] S15, jointly uses the prompt regularization loss and the contrastive learning loss to update the parameters of the gating network and the prompt vector until the parameters converge.

[0069] In a specific implementation of the present invention, the gating network is implemented using a model based on the Transformer architecture, whose function is to parse the text representation of the input query text and dynamically determine the weight contribution of each expert model. As the input of the gating network, the result generated by the Transformer structure of the gating network is processed by softmax to generate a probability distribution, and the fusion weight is output, including the content weight , style weight and content-style joint weight , content weight and style weights Used to weight the expert output features to obtain fusion features .

[0070] like Figure 4 As shown, in order to enhance task discriminability, the prompt word enhancement vector is introduced , from the prompt vector The process of the gating network selecting the matching prompt is expressed as:

[0071]

[0072] in, are trainable vectors, which are defined as content hint vector, style hint vector, and content-style joint hint vector. In this embodiment, the dimension of the hint vector is 218, which is the same as the dimension of the fusion feature; A hint vector representing the final choice; Indicates content weight , style weight and content-style joint weight The logic of the gating network to select the matching prompt is that when equal When selecting the content hint vector, equal Select the style hint vector when equal Selecting a joint content-style hint vector.

[0073] The gating network adaptively selects the corresponding prompt vector, enhances the model's ability to focus on the task, clarifies the differences between different tasks, and avoids task competition between expert models.

[0074] Hint Vector Initialization and update rules: are initialized to zero vectors. Under the content-style joint condition, since the training and reasoning paradigms are consistent, No update (keep zero vector), freeze The method forces the model to rely entirely on the weights dynamically generated by the gating network under joint conditions. and , rather than learning fixed joint prompts, to avoid the model directly "memorizing" the joint conditional features instead of truly understanding the independent semantics of content and style. This can enhance the modeling ability of content / style independent features and avoid over-parameterization of joint prompts.

[0075] The present invention is based on The corresponding prompt vector is selected based on the result to enhance the focusing ability of the model, and the selected prompt vector is added to the fusion feature for the next step of matching. To avoid overfitting of the prompt, the prompt regularization loss is introduced:

[0076]

[0077]

[0078] in, represents the prompt regularization loss, represents the style hint vector, represents the i-th dimension in the style hint vector, represents the content hint vector, represents the i-th dimension in the content hint vector, D represents the dimension of the hint vector, represents the regularization term, represents the scaling factor, represents the L1 norm, Represents the square of the L2 norm. In the expression of the prompt regularization loss, the first term encourages the difference between content cues and style cues, and the second term is the regularization term. The scaling factor controls the range of the regularization term, stabilizing the model and reducing the risk of overfitting.

[0079] In a specific implementation of the present invention, the training set data is implemented based on the WikiArt art style dataset and the COCO dataset. Figure 6 The training data construction process is demonstrated. Using a style transfer approach, images with explicit content labels from the COCO dataset are selected as content reference images, and images with explicit style labels from the WikiArt dataset are selected as style reference images. The InST style transfer model is then used to perform style transfer on the content images, generating synthetic images that contain both explicit content and style information. Each generated image is labeled with both content and style categories. After construction, images where the subject and label do not match are removed, such as images labeled "apple" where the apple occupies only a small area of the image. This ensures a balanced sample size for each content and style category. A total of 21,600 high-quality images are generated, encompassing 27 styles and 80 content categories. K-means clustering is performed on the style and content features, and the results show that the intra-class distance is significantly smaller than the inter-class distance, verifying the dataset's class discrimination and annotation consistency.

[0080] The content expert model used to extract image features adopts the ImageBind model, the style expert model adopts the InternVL model, and the text encoder used to extract text representation adopts the T5 text encoder. The ImageBind model, InternVL model, and T5 text encoder are all pre-trained models.

[0081] (2) Reasoning stage

[0082] like Figure 3 As shown in the figure, the query text generates text features through the text encoder, and the gated network determines the retrieval mode based on the text features. In the content retrieval mode and style retrieval mode, the fusion features output by the hybrid expert system are the image features output by the corresponding expert model. In the content-style joint retrieval mode, the fusion features output by the hybrid expert system are weighted image features. Image retrieval is performed in the corresponding database under different retrieval modes. It should be noted that Figure 3 The prompt part is omitted in . In fact, the addition result of the fusion feature and the corresponding prompt vector in the prompt word enhanced vector group is used as the image embedding vector for similarity retrieval.

[0083] In a specific implementation of the present invention, the process of searching using the trained model is as follows:

[0084] Step S21: Multimodal input and retrieval mode discrimination

[0085] User inputs query text and sample images , the gating network takes the text representation of the query text as input and generates content weights , style weight and content-style joint weight , automatically determines whether the search mode is content search, style search, or content-style combined search, and accesses the corresponding image database for similarity search. For example, if the query text is "Impressionist-style cats," the search mode is determined to be content-style combined search based on the intent, and the content-style combined database is accessed. In addition, content databases and style databases are also included.

[0086] Different databases store different features of candidate images, such as the content database, which stores the fusion features of candidate images superimposed with content hint vectors ; Style database, which stores the fusion features of candidate images superimposed with style hint vectors ; Joint content-style database, which stores the fused features of candidate images , here The weights generated by the gating network are updated, The parameters generated in the above training phase are fixed in the inference phase. Similarly, the parameters of the gating network are also generated and fixed in the above training phase.

[0087] In this embodiment, the process of automatically distinguishing the retrieval mode is as follows: equal When the content retrieval mode is equal When is the style retrieval mode, equal It is a content-style joint retrieval mode.

[0088] Step S22: Dynamically select the expert model of the MOE system according to the retrieval mode to obtain the fusion features finally output by the MOE system :

[0089]

[0090] Step S23: Fusion features of the final output of the MOE system Introduce the hint word enhancement vector to obtain the embedding vector for query :

[0091]

[0092] Step S24: Based on the embedding vector used for the query Perform similarity retrieval in the corresponding database, return the recommended image set that best matches the user input based on cosine similarity sorting, and give a matching score.

[0093] Figure 5Three typical user demand scenarios are presented: searching for image content only, searching for image style only, and searching for both image content and style. This invention, referred to as CCSR, allows users to express their search needs through text and example images. This invention accurately matches the image database based on the input intent, demonstrating the advantages of cross-modal and conditional search.

[0094] This example verifies the advantages of the present invention in content retrieval, style retrieval, and content-style retrieval tasks.

[0095] Table 1 shows a performance comparison for style retrieval and content retrieval tasks. Our CCSR method achieves the best performance in all four metrics, demonstrating its exceptional performance on a single task. Average precision @k represents the ranking quality of relevant results within the top k search results; recall @k indicates the proportion of "relevant" items captured within the top k returned results. Higher values for both metrics are preferred.

[0096] Table 1: Performance comparison of different models in style retrieval and content retrieval tasks

[0097]

[0098] Table 2 shows the main experimental results for content-style conditional retrieval, evaluating the model's performance under "joint conditions" (including both content and style). Our method again significantly outperforms other models, demonstrating that its multi-condition fusion capability is significantly superior to traditional models.

[0099] Table 2: Comparison of content-style joint retrieval performance of different models

[0100]

[0101] Figure 7 This figure compares the results of our proposed model with a typical retrieval model. The figure randomly selects several query examples and compares the performance of each model for simultaneous content and style retrieval. A green checkmark indicates an accurate retrieval result, meaning the model successfully matched a target image that met the query criteria in both content and style. A red cross indicates a retrieval failure, meaning the returned result does not match the user-specified content or style. This figure provides an intuitive understanding of the advantages and performance of our model in real-world applications, validating the effectiveness and advancement of our approach.

[0102] Figure 8This figure is a schematic diagram of the perceptual evaluation results of different retrieval models by experts in the field of art. This evaluation experiment invited professional experts in the field of art to participate. Based on the cross-modal conditional retrieval task proposed by this invention, the actual accuracy and user satisfaction scores of the retrieval results of different models were evaluated. The horizontal axis in the figure represents each model method, and the vertical axis represents the accuracy score of the expert evaluation. The CCSR model proposed by this invention received the highest score in the evaluation, significantly ahead of other methods, reflecting the outstanding advantages of this invention in matching practical applications with the needs of professional users.

[0103] Table 3 shows the results of the ablation experiment. The present invention analyzes the performance degradation of the average precision @k and recall @k after removing the hint learning strategy, contrast learning strategy and regularization strategy.

[0104] Table 3: Results of ablation experiments

[0105]

[0106] Based on the same inventive concept, the present invention also proposes a cross-modal conditional controllable retrieval system based on a hybrid expert system, the system comprising:

[0107] A dataset acquisition module is used to obtain an image training dataset with style labels and content labels;

[0108] A hybrid expert system based on a gated network consists of a gated network, a pre-trained content expert model, and a pre-trained style expert model. The gated network is used to output fusion weights. During the training phase, the fusion features output by the hybrid expert system are weighted image features. During the retrieval phase, the retrieval mode is first determined based on the query text input by the user. In the content retrieval mode and the style retrieval mode, the fusion features output by the hybrid expert system are the image features output by the corresponding expert models. In the content-style joint retrieval mode, the fusion features output by the hybrid expert system are weighted image features. The weighted image features refer to the weighted result of the fusion weights output by the gated network and the content image features and style image features.

[0109] A cue word enhancement module includes a learnable cue word enhancement vector group, and the sum of the fusion features output by the hybrid expert system and the corresponding cue vectors in the cue word enhancement vector group is used as the image embedding vector for similarity retrieval;

[0110] A training module is used to construct sample pairs based on the image training dataset, and jointly train the hybrid expert system based on the gated network and the cue word enhanced vector group using contrastive learning loss and cue regularization loss;

[0111] The retrieval module performs similarity retrieval in the corresponding database based on the image embedding vector used for the query, and returns a set of recommended images that best matches the user input based on cosine similarity sorting.

[0112] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Ordinary technicians in this field can understand and implement it without paying any creative work.

[0113] Embodiments of the system of the present invention can be applied to any device with data processing capabilities, such as a computer or other device. System embodiments can be implemented through software, hardware, or a combination of software and hardware. For example, a software implementation, as a logical device, is implemented by a processor of any device with data processing capabilities, reading corresponding computer program instructions from non-volatile memory into internal memory and executing them.

[0114] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they are not to be construed as limiting the scope of the present invention. Persons skilled in the art will readily appreciate that variations and modifications may be made without departing from the spirit of the present invention, and all such variations and modifications fall within the scope of protection of the present invention.

Claims

1. A cross-modal conditional controllable retrieval method based on a hybrid expert system for image retrieval, characterized in that: include: Obtain an image training dataset with style and content labels; Sample pairs are constructed based on the image training dataset. A hybrid expert system based on a gated network and a cue word enhancement vector group are trained using a joint contrastive learning loss and a cue regularization loss. The fused features output by the hybrid expert system are added to the corresponding cue vectors in the cue word enhancement vector group, and the added result is used as the image embedding vector for similarity retrieval. The cue word enhancement vector group includes a content cue vector, a style cue vector, and a content-style joint cue vector. The three cue vectors correspond to three retrieval modes respectively, and the initialization dimension of the cue vector is the same as the fused feature dimension. During the training phase, the fusion features output by the hybrid expert system are weighted image features; In the retrieval stage, the retrieval mode is first determined based on the query text entered by the user. In the content retrieval mode and style retrieval mode, the fusion features output by the hybrid expert system are the image features output by the corresponding expert model. In the content-style joint retrieval mode, the fusion features output by the hybrid expert system are weighted image features. The hybrid expert system includes a pre-trained content expert model and a pre-trained style expert model for generating content image features and style image features respectively; the weighted image features refer to the weighted results of the fusion weight output by the gated network and the content image features and style image features.

2. The cross-modal conditional controllable retrieval method based on a hybrid expert system according to claim 1, characterized in that: The joint content-style cue vector is fixed and does not participate in training.

3. The cross-modal conditional controllable retrieval method based on a hybrid expert system according to claim 1, characterized in that: The first term of the hint regularization loss is the negative value of the L1 norm of the style hint vector and the content hint vector, and the second term is a regularization term representing the sum of the squares of the L2 norms of each dimension of the style hint vector and the content hint vector.

4. The cross-modal conditional controllable retrieval method based on a hybrid expert system according to claim 1, characterized in that: The gating network takes the text features of the query text as input and generates fusion weights based on probability distribution; the fusion weights include content weights, style weights and content-style joint weights.

5. The cross-modal conditional controllable retrieval method based on a hybrid expert system according to claim 4, characterized in that: The gating network is composed of an encoder based on a Transformer structure and a softmax function. The encoder takes the text features of the query text as input to generate encoding features, and the softmax function processes the encoding features into probability distribution.

6. The cross-modal conditional controllable retrieval method based on a hybrid expert system according to claim 4, characterized in that: The corresponding prompt vector in the prompt word enhanced vector group is determined according to the fusion weight output by the gated network, and the prompt vector of the retrieval mode corresponding to the maximum value of the fusion weight is taken.

7. The cross-modal conditional controllable retrieval method based on a hybrid expert system according to claim 4, characterized in that: During the training phase, the query text of each sample pair is randomly selected from content query text, style query text, or content-style joint query text. If the sample pair matches the query text, the sample pair label is a positive label, otherwise it is a negative label.

8. The cross-modal conditional controllable retrieval method based on a hybrid expert system according to claim 1, characterized in that: Three databases are constructed during the retrieval phase: A content database, applied to the content retrieval mode, stores the content image features of candidate images superimposed with content hint vectors; The style database is used in the style retrieval mode, which stores the style image features of the candidate images after superimposing the style hint vector; The content-style joint database is applied to the content-style joint retrieval mode, which stores the weighted image features of candidate images. The weighted fusion weights are generated through a gating network.

9. A cross-modal conditional controllable retrieval system based on a hybrid expert system, used to implement the cross-modal conditional controllable retrieval method according to claim 1, characterized in that: The system comprises: A dataset acquisition module is used to obtain an image training dataset with style labels and content labels; A hybrid expert system based on a gated network consists of a gated network, a pre-trained content expert model, and a pre-trained style expert model. The gated network is used to output fusion weights. During the training phase, the fusion features output by the hybrid expert system are weighted image features. During the retrieval phase, the retrieval mode is first determined based on the query text input by the user. In the content retrieval mode and the style retrieval mode, the fusion features output by the hybrid expert system are the image features output by the corresponding expert models. In the content-style joint retrieval mode, the fusion features output by the hybrid expert system are weighted image features. The weighted image features refer to the weighted result of the fusion weights output by the gated network and the content image features and style image features. A cue word enhancement module includes a learnable cue word enhancement vector group. The fusion feature output by the hybrid expert system is added to the corresponding cue vector in the cue word enhancement vector group. The addition result is used as the image embedding vector for similarity retrieval. A training module is used to construct sample pairs based on the image training dataset, and jointly train the hybrid expert system based on the gated network and the cue word enhanced vector group using contrastive learning loss and cue regularization loss; The retrieval module performs similarity retrieval in the corresponding database based on the image embedding vector used for the query, and returns a set of recommended images that best matches the user input based on cosine similarity sorting.

Images