Dynamic deep multi-modal search method and device based on user intention perception, storage medium and computer equipment

Abstract

The application relates to the technical field of intelligent search, and discloses a dynamic deep multi-modal search method and device based on user intention perception, a storage medium and computer equipment, the method comprising the following steps: when a query request is received, first determining the complexity of a task and a professional field to which the task belongs, so as to determine an intention level and a query field; then activating the number of layers and the number of channels of text and visual encoders as required; meanwhile, extracting professional knowledge related to the query request in a field knowledge graph and converting the professional knowledge into natural language, and splicing the natural language with original query text to form an enhanced text; finally, encoding and fusing the enhanced text and a query image respectively, generating a query vector to complete result search. The method can be applied to the scenes of financial technology and medical health.

Description

Technical Field

[0001] This application relates to the field of intelligent search technology, and in particular to a dynamic deep multimodal search method, apparatus, storage medium and computer device based on user intent perception. Background Technology

[0002] In today's era of digital information explosion, users face massive and diverse data resources, making the demand for efficient and accurate searches increasingly urgent. For example, in healthcare scenarios, patients may upload medical images (such as X-rays and CT scans) and input symptom descriptions, hoping the search system can accurately pinpoint the disease type, recommend treatment plans, or link authoritative medical literature. In the fintech field, users may combine screenshots of financial statements, market dynamics text, and industry keywords to search for company risk assessments, investment opportunities, or policy interpretations. Multimodal search technology has emerged to meet this need, capable of comprehensively processing data from multiple modalities such as text and images, providing users with more comprehensive and accurate search results. However, different users initiate queries with complex and diverse intentions, and their requirements for the depth and precision of search results vary. For example, in professional field queries, users often expect to obtain more in-depth and detailed information; while in general queries, quickly obtaining general relevant information is sufficient.

[0003] However, existing multimodal models typically employ a static computation path during inference, meaning that regardless of the complexity or intent of the query request, the model activates all network layers and performs full computation. For simple query intents, this uniform approach results in significant waste of computational resources and inference delays; while for complex query intents, a fixed network depth may be insufficient to capture deep semantic relationships, leading to limited search accuracy and a lack of a mechanism to dynamically adjust the model's computational depth based on user intent.

[0004] Therefore, how to dynamically adjust search strategies based on user intent to better meet diverse user needs has become a key issue that urgently needs to be addressed in the field of multimodal search. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method, apparatus, storage medium, and computer device for dynamic deep multimodal search based on user intent awareness.

[0006] According to one aspect of this application, a dynamic deep multimodal search method based on user intent awareness is provided, the method comprising: In response to receiving a query request, the intent of the query request is identified to determine the intent level and query domain of the query request; Obtain the depth configuration and global sparsity threshold corresponding to the intent level, activate the corresponding layers of the text encoder and visual encoder according to the depth configuration, and perform structured sparse masking on the channels of the visual encoder whose channel importance is lower than the global sparsity threshold. Obtain a knowledge graph matching the query request from the domain knowledge graph library corresponding to the query domain, convert the knowledge graph into natural language description text as the enhanced text of the query request, and concatenate the enhanced text after the query text of the query request to obtain the query enhanced text; The text encoder encodes the query enhanced text into a text vector, the visual encoder encodes the query image in the query request into an image vector, the text vector and the image vector are fused to obtain a query vector, and the query results are searched based on the query vector.

[0007] According to another aspect of this application, a user intent-aware dynamic deep multimodal search device is provided, the device comprising: An intent recognition module is used to respond to a received query request by performing intent recognition on the query request and determining the intent level and query domain of the query request. The depth masking module is used to obtain the depth configuration and global sparsity threshold corresponding to the intent level, activate the corresponding layers of the text encoder and visual encoder according to the depth configuration, and perform structured sparse masking on the channels of the visual encoder whose channel importance is lower than the global sparsity threshold. The text enhancement module is used to obtain a knowledge graph that matches the query request from the domain knowledge graph library corresponding to the query domain, convert the knowledge graph into natural language description text as the enhanced text of the query request, and concatenate the enhanced text after the query text of the query request to obtain the query enhanced text; The result search module is used to encode the query enhanced text into a text vector using the text encoder, encode the query image in the query request into an image vector using the visual encoder, fuse the text vector and the image vector to obtain a query vector, and perform query result search based on the query vector.

[0008] According to another aspect of this application, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described dynamic deep multimodal search method based on user intent awareness.

[0009] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described dynamic deep multimodal search method based on user intent awareness.

[0010] By employing the above technical solutions, this application provides a user intent-aware dynamic deep multimodal search method, apparatus, storage medium, and computer device. Upon receiving a query request, the method first determines the complexity of the task and its relevant professional domain to establish the intent level and query domain. Then, it activates the layers and channels of the text and visual encoders as needed. Simultaneously, it extracts relevant professional knowledge from the domain knowledge graph and converts it into natural language, concatenating it with the original query text to form enhanced text. Finally, it encodes and fuses the enhanced text and query image to generate a query vector for result searching. This application constructs an intent-aware, driven dynamic depth control mechanism, flexibly adjusting the model's computational input based on the actual complexity of the user query, avoiding the resource waste caused by full model loading in traditional solutions. Through real-time injection of the domain knowledge graph, it semantically completes and improves the user's original query statement, providing rich external knowledge guidance to the model and compensating for the model's shortcomings in understanding professional terminology.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 The diagram illustrates a flowchart of a user intent-aware dynamic deep multimodal search method provided in an embodiment of this application. Figure 2 This illustration shows a flowchart of another user intent-aware dynamic deep multimodal search method provided in an embodiment of this application; Figure 3 A schematic diagram of the structure of a dynamic deep multimodal search device based on user intent awareness provided in an embodiment of this application is shown. Figure 4 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0013] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0014] This embodiment provides a user intent-aware dynamic deep multimodal search method, such as... Figure 1 As shown, the method includes: Step 101: In response to receiving a query request, perform intent recognition on the query request to determine the intent level and query domain of the query request.

[0015] Intent level refers to a hierarchical classification of the complexity of user query requests. For example, tasks can be divided into three levels—simple, medium, and complex—based on the number of modalities involved, the length of the logical chain, and the depth of expertise. This classification mechanism allows the model to predict the scale of computing resources required before task execution. Query domain refers to the professional vertical category to which the user's query content belongs, such as healthcare, finance and law, or general knowledge.

[0016] Specifically, upon receiving a user's query request, a lightweight intent recognition model is activated to quickly perform semantic parsing of the query content. During operation, this predictive model first extracts keyword signals and modal markers from the query text. For example, Level 1 (simple level): such as "find Paris weather pictures"; Level 2 (intermediate level): such as "compare iPhone 15 and Samsung Galaxy S24 camera specifications"; Level 3 (complex level): such as "analyze the relationship between lung cancer CT images and gene mutations". Simultaneously, the predictive model determines the query domain based on the terminology distribution characteristics of the query text. For example, when the query contains medical terminology clusters such as "nodule boundary," "gene mutation," and "chemotherapy regimen," the query domain is marked as healthcare; when the query contains financial terminology clusters such as "merger and acquisition terms," ​​"secondary market," and "non-performing assets," it is marked as fintech. When the predictive model encounters anomalies such as ambiguous terminology distribution or inability to clearly determine the domain, a fallback mechanism is automatically triggered to revert the query domain to a general domain, preventing subsequent knowledge graph loading errors due to domain misjudgment.

[0017] It should be noted that the aforementioned lightweight intent recognition model refers to a lightweight predictive neural network deployed at the search entry point, achieving rapid classification by processing only the shallow semantic features of the query text. During the model training phase, a multi-type query sample set, encompassing factual queries, comparative queries, and inferential queries, is first collected from historical search logs. Each sample is assigned an intent level label and a domain label manually or using rule-based automatic annotation tools, forming a supervised training dataset. The labeled query texts are then fed into the intent recognition model one by one. After passing through a word embedding layer and the first few Transformer encoding layers, the model outputs the probability distribution of the intent level and the probability distribution of the domain label at the classification head. The cross-entropy loss between the probability distribution output by the classification head and the true labels is calculated, and the weighted sum of the intent level classification loss and the domain classification loss is used as the total training loss. The model parameters are iteratively updated using the backpropagation algorithm until the total loss converges to a stable level on the validation set.

[0018] Step 102: Obtain the depth configuration and global sparsity threshold corresponding to the intent level, activate the corresponding layers of the text encoder and visual encoder according to the depth configuration, and perform structured sparse masking on the channels of the visual encoder whose channel importance is lower than the global sparsity threshold.

[0019] Among them, depth configuration refers to the number of layers actually involved in the computation in the model encoder. Its configuration range is mapped to the intent level. The simple level corresponds to the shallow configuration, which only activates the basic feature extraction layer at the front end of the encoder. The complex level corresponds to the deep configuration, which activates all layers of the encoder to support deep semantic reasoning. This flexible configuration allows the same model to adapt to query tasks of different difficulties. The global sparsity threshold is a unified critical value used to determine whether each channel in the neural network is activated. When the importance score of a channel is lower than the threshold, the channel is marked as dormant, its output is set to zero, and subsequent computations that depend on the channel are skipped. The structured sparse mask refers to masking the corresponding channels so that they do not participate in the computation.

[0020] Specifically, after obtaining the intent level, the corresponding depth configuration parameters and global sparsity threshold are read from a preset mapping table. Then, activation operations are performed on the text encoder and visual encoder according to the depth configuration parameters: for layers configured to be activated, they continue to participate in forward propagation computation; for deeper layers exceeding the configuration range, their computational units are placed in a dormant state, and the data flow is truncated before reaching these layers. After layer activation, channel-level pruning is performed within the activated layers: the output channels of each convolutional or fully connected layer are scored using a pre-trained importance scorer, and channels with scores below the global sparsity threshold are subjected to a structured sparse mask, forcing their output values ​​to zero. For example, in a medical scenario, when processing a simple retrieval request for a chest CT image, the visual encoder only activates the front-end basic texture layers, and about 70% of the channels in these layers are masked and dormant because they are below the threshold; while in a financial scenario, when processing a merger and acquisition contract text involving multiple nested clauses, the text encoder activates the middle semantic reasoning layer and retains most channels to maintain logical analysis capabilities. Furthermore, when the device temperature exceeds a preset safety value during execution, the global sparsity threshold can be automatically increased, causing more channels to enter a sleep state to reduce computational power consumption and prevent chip overheating. Through a two-level pruning mechanism of hierarchical activation and channel masking, the computational scale of the model and the query difficulty are dynamically matched, reducing memory usage and power consumption while ensuring semantic analysis accuracy.

[0021] It should be noted that during the pre-training or fine-tuning phase of the visual encoder, a learnable scoring vector of equal length to the number of channels is initialized for each convolutional layer that supports dynamic pruning. In each forward propagation, the output values ​​of each channel are not processed additionally; the scoring vector is used only as an independent parameter in subsequent gradient calculations. During the inference phase, the visual encoder no longer updates the scoring vector; it is only used as a basis for decision-making.

[0022] Step 103: Obtain the knowledge graph matching the query request from the domain knowledge graph library corresponding to the query domain, convert the knowledge graph into natural language description text as the enhanced text of the query request, and concatenate the enhanced text after the query text of the query request to obtain the query enhanced text.

[0023] Among them, the domain knowledge graph library refers to a structured knowledge storage system built according to professional fields. Its internal structure consists of triplet networks with entities as nodes and attributes and relationships as edges. For example, the medical domain graph stores the entity "pulmonary nodule" and its relationship with the attribute "ground-glass opacity" and the entity "EGFR mutation". The financial domain graph stores the entity "convertible bond" and its relationship with the attribute "conversion price" and the legal entity "Measures for the Administration of Acquisitions of Listed Companies". This library enables general search models to obtain background knowledge of professional fields. Enhanced text refers to the supplementary context formed by converting the structured relationships retrieved from the knowledge graph into coherent natural language descriptions. Its role is to provide the professional constraint information required for disambiguation of the original query and solve the problem of mismatch caused by terminology ambiguity in cross-modal retrieval.

[0024] Specifically, based on the query domain tags, subgraph extraction is performed from the corresponding domain knowledge graph library. The extraction process starts with named entities in the query text as the starting node, expanding along relation edges to neighboring nodes within a two-hop range, collecting the obtained entity, relation, and attribute triples into a subgraph. Then, the graph text conversion module is activated, converting each triple in the subgraph into a natural language statement. For example, in the medical field, the triple "pulmonary nodule - requires differentiation - inflammatory pseudotumor" is converted to "In clinical diagnosis, pulmonary nodules need to be differentiated from inflammatory pseudotumors," and in the financial field, the triple "convertible bond - includes - put option" is converted to "Convertible bonds typically include a put option, giving investors the right to sell the bond back to the issuer under specific conditions." The generated enhanced text is then appended to the original query text to form the query enhanced text. Through knowledge graph retrieval and text injection mechanisms, the originally domain-deficient general model gains contextual support from professional knowledge.

[0025] Step 104: Encode the query enhancement text into a text vector using the text encoder, encode the query image in the query request into an image vector using the visual encoder, fuse the text vector and the image vector to obtain a query vector, and perform a query result search based on the query vector.

[0026] Specifically, the query-enhanced text is input into a text encoder, and after multi-layer self-attention transformation, the hidden vector corresponding to the global position in the output layer is extracted as the text vector. If the query request includes image data, the user-uploaded query image is input into a visual encoder, and after block embedding and layer-by-layer encoding, its global representation vector is extracted as the image vector. For mixed text and image queries, the text vector and image vector are weighted and fused to obtain the query vector. For pure text queries, the query vector is directly taken from the text vector; for pure image queries, the query vector is directly taken from the image vector. Subsequently, using the query vector as the retrieval key, a nearest neighbor search is performed in a preset image feature index library, returning the candidate result set with the highest cosine similarity to the query vector. Through dual-tower encoding and fusion retrieval mechanism, a unified semantic expression of information from different modalities is achieved, enabling mixed text and image queries to be matched in a unified vector space.

[0027] By applying the technical solution of this embodiment, upon receiving a query, the complexity of the task and its professional domain are first determined. Then, the number of layers and channels of the text and visual encoders are activated as needed. Simultaneously, relevant professional knowledge is extracted from the domain knowledge graph and converted into natural language context for injection into the query. By constructing an intent-aware, dynamic depth control mechanism, the computational input of the model is flexibly adjusted according to the actual complexity of the user's query, avoiding the resource waste caused by full model loading in traditional solutions. Real-time injection from the domain knowledge graph compensates for the shortcomings of general models in understanding professional terminology.

[0028] In one optional embodiment, intent recognition is performed on the query request to determine the intent level and query domain of the query request, including: intent recognition is performed on the query request to determine the intent level, query domain, and intent entropy value corresponding to the intent level; the text weight of the text vector is determined based on the intent entropy value; correspondingly, the text vector and the image vector are fused to obtain a query vector, including: determining the image weight of the image vector based on the text weight, and performing a weighted fusion calculation on the text vector and the image vector based on the text weight and the image weight to obtain the query vector.

[0029] Among them, intent entropy is a quantitative indicator that measures the degree of disorder in the confidence distribution of intent classifier output. When a high degree of confidence is given for a certain intent level, the entropy value approaches zero. This indicator enables the system to perceive the ambiguity of the user query itself. Text weight is the proportion of the text vector in the final fused query vector. Its value is dynamically adjusted by intent entropy. When the user's text description is detailed and clear, the text weight automatically increases. When the text is ambiguous or missing, the text weight decreases accordingly.

[0030] Specifically, during the intent recognition process, the lightweight intent recognition model outputs a softmax probability distribution vector in addition to the intent level and query domain label. The three elements of this distribution vector represent the decision confidence levels for the simple, medium, and complex levels, respectively. The entropy value of this distribution is then calculated as the intent entropy value, calculated by taking the logarithm of each of the three confidence levels, weighted summing, and then taking the negative value. After obtaining the intent entropy value, the text weight is determined based on the preset mapping relationship between entropy value intervals and weights. When the intent entropy value is in the low interval, it indicates that the user's query wording is precise and the intent is clear; the text weight is set to a high value, and the fusion vector is dominated by text semantics. When the intent entropy value is in the medium interval, it indicates that the query has some ambiguity; the text weight and image weight tend to be balanced. When the intent entropy value is in the high interval, it indicates that it is difficult to determine the user's true intent. The image weight is automatically obtained by subtracting the text weight from a constant value. In a healthcare scenario, when a doctor inputs "find confirmed EGFR mutation cases with ground-glass nodules similar in morphology to those in this CT scan," the lightweight intent recognition model gives high confidence in the complexity level, with a low entropy value. The text weight is set to 0.7, the image weight to 0.3, and the fusion vector emphasizes the disease concept and genetic information described in the text. In a fintech scenario, when an analyst uploads a screenshot of a merger agreement and inputs "find clauses containing similar control change triggering performance-based earnouts," the lightweight intent recognition model considers it complex, with a low entropy value, and the text weight dominates the fusion vector.

[0031] In one optional embodiment, before searching for query results based on the query vector, the method further includes: obtaining a projection matrix corresponding to the query domain; wherein the projection matrix is ​​pre-trained using image-text pairs within the query domain, and the text vector is mapped to a visual feature space based on the projection matrix to obtain a visual spatial text vector corresponding to the text vector; correspondingly, searching for query results based on the query vector includes: performing data retrieval based on the query vector to obtain multiple candidate images, encoding the candidate images into candidate image vectors using the visual encoder; calculating the similarity between the visual spatial text vector and the candidate image vectors, sorting the candidate images based on the similarity, and outputting query results based on the sorted candidate images.

[0032] Among them, the domain projection matrix refers to a linear transformation parameter matrix obtained by training with paired text and image samples within the domain. This matrix encodes the mapping rules between text descriptions and visual representations within a specific domain, so that the text vectors, after being transformed by it, can fall into a semantic subspace that is directly comparable to the image vectors. The visual space text vector refers to the high-dimensional vector representation obtained by left-multiplying the text vectors by the domain projection matrix. Although this vector originates from the text encoder, it has distribution characteristics consistent with image features. The candidate image vector refers to the global representation vector extracted by the visual encoder from multiple candidate images returned during the recall stage.

[0033] Specifically, this step is performed after query vector generation and before the final search results are output. First, based on the query domain label, a pre-trained domain projection matrix is ​​retrieved from the projection matrix parameter library. The parameter library is organized by domain: the medical domain stores the medical projection matrix, which, during training, learned to map medical terms such as lung nodules, skin lesions, and pathological grading to the visual feature representations of corresponding image regions; the financial domain stores the financial projection matrix, which learned to map financial terms such as convertible bonds, M&A terms, and performance-based agreements to the visual feature representations of contract layout and signature arrangement. If the query domain label points to medical, the medical projection matrix is ​​retrieved; if it points to financial, the financial projection matrix is ​​retrieved; if the domain label is general or cannot be clearly determined, a general projection matrix pre-trained on mixed multi-domain data is retrieved as a fallback. After retrieval, a linear mapping is performed on the text vector output by the text encoder, multiplying the text vector as a row vector by the projection matrix on the left, resulting in the visual space text vector. Then, a two-stage retrieval process begins. The first stage is data recall. Using the fused query vector as the retrieval key, an approximate nearest neighbor search is performed in a pre-built image feature index. The index stores offline pre-encoded visual feature vectors of a massive number of candidate images. Recall is used to quickly filter out a batch of candidate images whose cosine similarity to the query vector exceeds a preset recall threshold. The second stage is fine sorting. Each candidate image obtained in the recall stage is mapped to its corresponding visual feature vector. If the offline pre-encoded result is already stored in the index, it is read directly; otherwise, it is extracted in real-time by a visual encoder. Then, the cosine similarity between the visual space text vector and each candidate image vector is calculated. After calculating the similarity of all candidate images, they are sorted from highest to lowest score, and a preset number of the top-ranked results are taken as the final retrieval output.

[0034] Taking a healthcare scenario as an example, a doctor submits a query text "Lung CT shows ground-glass nodules with lobulation." After the text vector is mapped to a visual space text vector through a medical projection matrix, this vector already carries the visual space features corresponding to the ground-glass texture and lobulated contour. In the recall phase, multiple candidate CT images containing various nodule morphologies are quickly filtered from the index. In the fine-tuning phase, the similarity between the candidate image vector and the visual space text vector is compared one by one, thereby outputting an accurate image report.

[0035] Through the domain projection matrix mapping and two-stage ranking mechanism in this embodiment, the text vector is transferred from the text semantic space to the visual feature space, eliminating the bias that occurs during direct cross-modal comparison. The two-stage architecture uses a recall layer to ensure retrieval speed and a fine ranking layer to inject domain knowledge constraints to achieve high-precision semantic matching.

[0036] In one optional embodiment, the training process of the projection matrix corresponding to the query domain includes: Retrieve one image-text pair sample from multiple image-text pair samples within the query domain, such as... Figure 2 As shown, perform the following projection matrix update operation: Step 201: Encode the text samples in the image-text pair samples into text sample vectors using the text encoder, and encode the image samples in the image-text pair samples into image sample vectors using the visual encoder. Step 202: Map the text sample vector into the visual space according to the initial projection matrix to obtain the visual space text sample vector corresponding to the text sample vector; Step 203: Calculate the sample vector similarity between the visual space text sample vector and the image sample vector; determine the similarity loss term based on the sample vector similarity; perform regularization calculation on the initialized projection matrix to obtain the projection matrix regularization term; calculate the product of the projection matrix regularization term and the preset regularization coefficient as the projection matrix regularization loss term; and determine the projection loss based on the similarity loss term and the projection matrix regularization loss term. Step 204: Update the initial projection matrix according to the projection loss, return to the step of obtaining a text-image pair sample from multiple text-image pair samples in the query domain, obtain the next text-image pair sample, perform the projection matrix update operation, until the obtained projection matrix satisfies the preset convergence condition.

[0037] In this embodiment, the training process of the domain projection matrix involves constructing an objective function jointly optimized by a similarity loss term and a regularization loss term. The projection matrix parameters are iteratively updated on paired text-image samples, allowing it to gradually learn the mapping relationship between text descriptions and visual representations within the domain. Here, a text-image pair refers to a training sample pair within the same domain consisting of mutually matching text descriptions and image data. For example, in the medical field, a radiology report text and its corresponding CT scan image constitute one sample; in the financial field, a contract terms summary and its corresponding signed page scan constitute one sample. The similarity loss term is an error metric measuring the semantic alignment between the projected text vector and the matching image vector. The projection matrix regularization term is a constraint penalty imposed on the projection matrix parameters.

[0038] Specifically, a dedicated projection matrix is ​​trained independently for each domain, using image-text aligned corpora within that domain as training data. First, the projection matrix parameters are randomly initialized. Then, samples are retrieved one by one from the image-text pair sample set within the query domain, and a projection matrix update operation is performed on each sample. The update process is as follows: text samples from the image-text pair samples are input into the text encoder, and their final layer global representation vectors are extracted as text sample vectors; image samples from the image-text pair samples are input into the visual encoder, and their global representation vectors are extracted as image sample vectors. Both the text encoder and visual encoder maintain parameter freezes during this training phase, with only the projection matrix participating in gradient updates. The text sample vectors are left-multiplied by the current projection matrix as row vectors to obtain the visual space text sample vectors. The cosine similarity between the visual space text sample vectors and image sample vectors is calculated, and this similarity is substituted into the cosine distance formula to obtain the similarity loss term. Simultaneously, the Frobenius norm is calculated for the current projection matrix, and the norm value is multiplied by a preset regularization coefficient to obtain the projection matrix regularization loss term. The similarity loss term and the regularization loss term are added to obtain the projection loss. Calculate the gradient of the projection loss with respect to each element of the projection matrix, and update the parameter values ​​of the projection matrix along the gradient descent direction. After the update is complete, return to retrieve the next image / text pair and continue the update operation until all samples in the training set have been traversed, completing one round of training. The training process continues for multiple rounds of iteration until the projection matrix meets the preset convergence condition, such as the change in projection loss between two adjacent training rounds being less than a preset minimum threshold.

[0039] In one optional embodiment, in response to receiving a query request, intent recognition is performed on the query request to determine the intent level and query domain of the query request, including: in response to receiving a query request, constructing a preprocessing task based on the query request, and executing the preprocessing task through an FPGA to realize intent recognition of the query request and determine the intent level and query domain of the query request; The method further includes: constructing a core computing task based on the intent level and query domain of the query request, and executing the core computing task through a GPU to achieve activation and masking of the text encoder and visual encoder, acquisition of the knowledge graph, encoding and fusion of text vectors and image vectors, and searching for query results; wherein, the process of executing the core computing task through the GPU includes: if there are remaining tokens in the pre-created token bucket, consuming one token to put the core computing task corresponding to the query request into the request buffer waiting queue, and performing resource scheduling based on the priority of tasks in the request buffer waiting queue, wherein the tokens in the token bucket are generated according to a dynamic token generation rate limited to a preset token bucket capacity, and the dynamic token generation rate is determined based on the current device hardware health value, the queue depth of the request buffer queue, and the current system load.

[0040] In this embodiment, a lightweight intent recognition model is embedded in an FPGA (Field-Programmable Gate Array). When a user query request arrives, the FPGA performs text segmentation, embedding lookup, and shallow attention computation in a pipelined manner on its customized computing array. Within a very short latency, it determines the intent level, intent entropy value, and query domain label, and sends the determination result as a control signal to the subsequent GPU scheduling module. Simultaneously, the FPGA performs text preprocessing operations, including stop word filtering and word embedding vector lookup. The preprocessed tensor data is written to a designated buffer in the GPU's video memory via a direct memory access channel with the GPU. Then, the GPU core computation stage begins. The GPU includes token buckets generated according to a dynamic token generation rate. The dynamic token generation rate is recalculated in each decision cycle, taking into account three factors: the current hardware health of the device (the rate is reduced when the core temperature exceeds a safe threshold); the current depth of the request buffer queue (the rate is reduced accordingly when the number of backlogged tasks in the queue continues to increase to prevent the queue from growing indefinitely); and the current overall system load, which is the product of the number of available GPU instances and the throughput of a single instance (the higher the load, the lower the rate). Before each query request's corresponding core computation task enters the GPU for execution, it first checks if there are any remaining tokens in the token counter. If there are remaining tokens, one token is consumed and the core computation task is placed at the tail of the request buffer waiting queue; if all tokens are exhausted, the task is placed in the waiting area until a token is generated. The request buffer waiting queue is a first-in, first-out (FIFO) structure. Resource scheduling is performed on each task in the order of tasks in the queue, allocating the smallest computing unit in the GPU multi-instance computing partition to the task, and sending the task to the allocated computing partition to perform a series of core computations, including layer activation and channel masking of text encoders and visual encoders, domain knowledge graph retrieval and text conversion, encoding and fusion of text vectors and image vectors, and final retrieval and ranking. This embodiment embeds lightweight intent recognition and text preprocessing into the FPGA, shortening the response time of query requests, while freeing up GPU computing resources for core deep learning inference.

[0041] In one optional embodiment, resource scheduling based on the task order in the request buffer waiting queue includes: performing resource scheduling for each core computing task sequentially according to the task order in the request buffer waiting queue, wherein resource scheduling for any core computing task is performed as follows: determining the estimated activation computational load of the core computing task based on the activation layer number of the text encoder corresponding to the core computing task, the activation layer number of the corresponding visual encoder, and the number of open channels; determining the minimum number of GPU instances required for the core computing task based on the estimated activation computational load and the computing power of a single GPU instance; and performing resource scheduling for the core computing task according to the minimum number of GPU instances required.

[0042] The computing power of a single GPU instance refers to the theoretical floating-point operation capability that the smallest independent computing unit divided from a physical graphics processor through multi-instance GPU technology can provide per unit time. This computing power value is the smallest granularity benchmark for resource scheduling. The minimum number of GPU instances required refers to the minimum number of GPU instances that need to be allocated to ensure that the core computing tasks are completed within the preset response deadline. It is calculated by dividing the estimated value of the active computing load by the computing power of a single GPU instance and rounding up to ensure that the task does not time out due to insufficient resources or waste due to excessive resources.

[0043] Once the core computing tasks enter the request buffer queue sequentially, the scheduler performs resource scheduling for each task one by one according to the first-in, first-out (FIFO) principle. For the currently processed core computing task, the depth configuration parameters are first extracted from the task, including the number of layers activated by the text encoder and the visual encoder, as well as the number of channels that remain active in each activated layer as output by the dynamic channel mask module. Then, the baseline computational load of each activated layer of the text and visual encoders is weighted and integrated with the corresponding layer's channel retention ratio, and the two results are added together to obtain the estimated activation computational load for this core computing task. After obtaining the estimated activation computational load, the theoretical computing power of a single GPU instance is read from the GPU resource management module. The estimated activation computational load is divided by this computing power to obtain the theoretical value of the required number of instances. This theoretical value is then rounded up to obtain the minimum required number of GPU instances. If the result is less than one, it is set to one, ensuring that at least one instance is allocated. This ensures that the GPU computing power obtained by each core computing task precisely matches its actual computational complexity. Subsequently, the corresponding number of available instances are allocated from the multi-instance computing partition of the GPU cluster and assigned to the task, establishing a binding relationship between the task and the instance. Once the task is completed, the occupied instance is released back to the public availability pool.

[0044] In one optional embodiment, the depth configuration corresponding to the intent level includes a layer range; activating the text encoder and visual encoder according to the depth configuration includes: obtaining the current device temperature value and the system effective throughput; determining the target activation layer in the layer range based on the intent entropy value corresponding to the intent level, the current device temperature value, and the system effective throughput; and activating the text encoder and visual encoder according to the depth configuration.

[0045] Among them, the layer range refers to the selectable range of activation layers corresponding to a certain intent level. Its lower limit is the minimum semantic analysis depth required for the task at that level, and its upper limit is the maximum computational depth that the task at that level can benefit from. The current device temperature value refers to the core temperature reading collected in real time by the out-of-band monitoring agent deployed on the GPU. This temperature reflects the chip's current heat dissipation load and thermal protection margin. When the temperature approaches the preset safety threshold, computational input should be suppressed to avoid performance degradation or hardware damage. The system effective throughput refers to the number of core computing tasks that the current system can successfully complete per unit time under the preset response deadline constraint.

[0046] Specifically, during layer activation, the layer range corresponding to the current intent level is retrieved from the depth configuration table. Subsequently, the current device temperature, system effective throughput, and intent entropy value are retrieved in parallel. After obtaining these three parameters, a dynamic decision is made regarding the target activation layer within the layer range. For example, the decision logic follows three principles working in tandem. Principle one is entropy-oriented: when the intent entropy value is in a low range, it indicates that the classifier is highly confident in determining the intent level, and the target activation layer moves towards the lower limit of the layer range because the determined task can be completed without redundant depth; when the intent entropy value is in a high range, it indicates that the query semantics are ambiguous or the classifier has difficulty judging, and the target activation layer moves towards the upper limit of the layer range to retain deeper reasoning capabilities to handle potentially complex needs. Principle two is temperature constraint: when the current device temperature exceeds the preset warning temperature, the target activation layer is forced to decrease towards the lower limit of the layer range to reduce computational power consumption and alleviate heat dissipation pressure; this constraint does not take effect when the temperature is within a safe range. Principle three is throughput compensation: When the system's effective throughput is lower than the preset low watermark, it indicates that the system's current processing capacity is strained and queuing pressure is increasing. The target activation layer number is appropriately reduced to shorten single-task execution time and improve overall throughput efficiency. This constraint does not take effect when the throughput is at a healthy level. After determining the target activation layer number, the scheduler issues activation commands to the text encoder and visual encoder, setting all layers from the first layer up to the target activation layer to an active state, while deeper layers exceeding the target activation layer remain dormant. Thus, when the equipment is healthy, the system can fully release its deep computing capabilities to handle complex queries, and when the equipment becomes hot or the load is too high, it automatically converges the computing scale to ensure system stability.

[0047] Taking a healthcare scenario as an example, when the system is processing a batch of routine image follow-up queries, the intent entropy value is generally low, the device temperature is within a safe range, the system throughput is healthy, the target activation layer is stable at a shallow configuration, the response is rapid, and the power consumption is controllable. When a large number of complex image analysis requests suddenly appear and the intent entropy value increases, the target activation layer automatically moves up to the upper limit of the layer range to provide sufficient semantic analysis depth. If continuous high-intensity computing causes the GPU temperature to approach the warning value, the temperature constraint is triggered first. Even if the intent entropy value is still high, the target activation layer is forcibly reduced to the middle of the range, ensuring hardware safety while retaining a certain analysis depth.

[0048] Furthermore, as Figure 1 In terms of specific implementation, this application provides a dynamic deep multimodal search device based on user intent awareness, such as... Figure 3 As shown, the device includes: An intent recognition module is used to respond to a received query request by performing intent recognition on the query request and determining the intent level and query domain of the query request. The depth masking module is used to obtain the depth configuration and global sparsity threshold corresponding to the intent level, activate the corresponding layers of the text encoder and visual encoder according to the depth configuration, and perform structured sparse masking on the channels of the visual encoder whose channel importance is lower than the global sparsity threshold. The text enhancement module is used to obtain a knowledge graph that matches the query request from the domain knowledge graph library corresponding to the query domain, convert the knowledge graph into natural language description text as the enhanced text of the query request, and concatenate the enhanced text after the query text of the query request to obtain the query enhanced text; The result search module is used to encode the query enhanced text into a text vector using the text encoder, encode the query image in the query request into an image vector using the visual encoder, fuse the text vector and the image vector to obtain a query vector, and perform query result search based on the query vector.

[0049] Optionally, the intent recognition module is specifically used to: perform intent recognition on the query request using a pre-trained intent recognition model, determine the intent level, query domain, and intent entropy value corresponding to the intent level of the query request; and determine the text weight of the text vector based on the intent entropy value. Accordingly, the result search module is specifically used to: determine the image weight of the image vector based on the text weight, and perform a weighted fusion calculation on the text vector and the image vector based on the text weight and the image weight to obtain the query vector.

[0050] Optionally, the result search module is further configured to: obtain a projection matrix corresponding to the query domain before performing a query result search based on the query vector; wherein the projection matrix is ​​pre-trained using image and text pairs within the query domain; and map the text vector to the visual feature space based on the projection matrix to obtain the visual space text vector corresponding to the text vector. Accordingly, the result search module is specifically used for: performing data retrieval based on the query vector to obtain multiple candidate images; encoding the candidate images into candidate image vectors using the visual encoder; calculating the similarity between the visual spatial text vector and the candidate image vectors; sorting the candidate images based on the similarity; and outputting query results based on the sorted candidate images.

[0051] Optionally, the training process of the projection matrix corresponding to the query domain includes: obtaining one image-text pair sample from multiple image-text pair samples in the query domain, and performing the following projection matrix update operation: encoding the text samples in the image-text pair sample into text sample vectors using the text encoder, and encoding the image samples in the image-text pair sample into image sample vectors using the visual encoder; mapping the text sample vectors into the visual space according to the initialized projection matrix to obtain the visual space text sample vectors corresponding to the text sample vectors; and calculating the sample vector phase between the visual space text sample vectors and the image sample vectors. Similarity is determined based on the similarity of the sample vectors. A regularization term is calculated on the initial projection matrix to obtain a projection matrix regularization term. The product of the projection matrix regularization term and a preset regularization coefficient is used as the projection matrix regularization loss term. The projection loss is determined based on the similarity loss term and the projection matrix regularization loss term. The initial projection matrix is ​​updated based on the projection loss. The process then returns to the step of obtaining a single image-text pair sample from multiple image-text pair samples within the query domain to obtain the next image-text pair sample and perform the projection matrix update operation until the obtained projection matrix satisfies the preset convergence condition.

[0052] Optionally, the intent recognition module is configured to, in response to receiving a query request, construct a preprocessing task based on the query request, and execute the preprocessing task through an FPGA to realize intent recognition of the query request, determine the intent level and query domain of the query request, and the deep masking module, the text enhancement module, and the result search module perform core computing tasks through a GPU to perform core computing, wherein the core computing includes activation and masking of text encoders and visual encoders, acquisition of knowledge graphs, encoding and fusion of text vectors and image vectors, and searching for query results; The process by which the depth mask module executes the core computing task through the GPU includes: if there are remaining tokens in the pre-created token bucket, then one token is consumed to put the core computing task corresponding to the query request into the request buffer waiting queue, and resource scheduling is performed based on the order of tasks in the request buffer waiting queue. The tokens in the token bucket are generated according to a dynamic token generation rate with a preset token bucket capacity as the limit. The dynamic token generation rate is determined based on the current device hardware health value, the queue depth of the request buffer queue, and the current system load.

[0053] Optionally, the depth masking module performs core calculations via a GPU, including: Resource scheduling is performed on each core computing task sequentially according to the order of tasks in the request buffer waiting queue, wherein resource scheduling for any core computing task is performed in the following manner: Based on the number of activation layers of the text encoder corresponding to the core computing task, the number of activation layers of the corresponding visual encoder, and the number of open channels, the estimated activation computational load of the core computing task is determined. Based on the estimated activation computational load and the computing power of a single GPU instance, the minimum number of GPU instances required for the core computing task is determined. Resource scheduling is performed for the core computing tasks according to the minimum required number of GPU instances.

[0054] Optionally, the depth configuration corresponding to the intent level includes a layer range; the depth mask module is specifically used for: obtaining the current device temperature value and the system effective throughput; determining the target activation layer in the layer range according to the intent entropy value corresponding to the intent level, the current device temperature value, and the system effective throughput; and activating the text encoder and visual encoder according to the depth configuration.

[0055] It should be noted that other corresponding descriptions of the functional units involved in the user intent-aware dynamic depth multimodal search device provided in this application embodiment can be found in the following references. Figures 1 to 2 The corresponding descriptions in the method will not be repeated here.

[0056] This application also provides a computer device, which may specifically be a personal computer, a server, a network device, etc. Figure 4As shown, the computer device includes a bus, a processor, memory, and a communication interface, and may also include an input / output interface and a display device. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores location information. The network interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0057] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0058] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0059] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0060] It should be noted that the user personal information involved in the embodiments of this application is all authorized (with the knowledge and consent) by the relevant parties or fully authorized by all parties, and the executing entity can obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with the relevant laws and regulations of the relevant countries and regions, and do not violate public order and good morals. It should be noted that if any software tools or components other than those of this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use.

[0061] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0062] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0063] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A dynamic deep multimodal search method based on user intent awareness, characterized in that, The method includes: In response to receiving a query request, the intent of the query request is identified to determine the intent level and query domain of the query request; Obtain the depth configuration and global sparsity threshold corresponding to the intent level, activate the corresponding layers of the text encoder and visual encoder according to the depth configuration, and perform structured sparse masking on the channels of the visual encoder whose channel importance is lower than the global sparsity threshold. Obtain a knowledge graph matching the query request from the domain knowledge graph library corresponding to the query domain, convert the knowledge graph into natural language description text as the enhanced text of the query request, and concatenate the enhanced text after the query text of the query request to obtain the query enhanced text; The text encoder encodes the query enhanced text into a text vector, the visual encoder encodes the query image in the query request into an image vector, the text vector and the image vector are fused to obtain a query vector, and the query results are searched based on the query vector.

2. The method according to claim 1, characterized in that, The query request is subjected to intent identification to determine its intent level and query domain, including: The intent of the query request is identified by a pre-trained intent recognition model, and the intent level, query domain, and intent entropy value corresponding to the intent level are determined. The text weight of the text vector is determined based on the intent entropy value; Accordingly, the query vector is obtained by fusing the text vector and the image vector, including: The image weight of the image vector is determined based on the text weight, and the text vector and the image vector are weighted and fused based on the text weight and the image weight to obtain the query vector.

3. The method according to claim 1, characterized in that, Before performing a search for query results based on the query vector, the method further includes: Obtain the projection matrix corresponding to the query domain; wherein, the projection matrix is ​​pre-trained using image and text pairs within the query domain; The text vector is mapped to the visual feature space based on the projection matrix to obtain the visual space text vector corresponding to the text vector; Accordingly, the search for query results based on the query vector includes: Data retrieval is performed based on the query vector to obtain multiple candidate images, and the candidate images are encoded into candidate image vectors by the visual encoder. Calculate the similarity between the visual space text vector and the candidate image vector, sort the candidate images based on the similarity, and output the query results based on the sorted candidate images.

4. The method according to claim 3, characterized in that, The training process for the projection matrix corresponding to the query domain includes: From the multiple image-text pairs within the query domain, obtain one image-text pair sample and perform the following projection matrix update operation: The text encoder encodes the text samples in the image-text pair into text sample vectors, and the visual encoder encodes the image samples in the image-text pair into image sample vectors. The text sample vector is mapped to the visual space according to the initial projection matrix to obtain the visual space text sample vector corresponding to the text sample vector; Calculate the sample vector similarity between the visual space text sample vector and the image sample vector, determine the similarity loss term based on the sample vector similarity, perform regularization calculation on the initial projection matrix to obtain the projection matrix regularization term, calculate the product of the projection matrix regularization term and the preset regularization coefficient as the projection matrix regularization loss term, and determine the projection loss based on the similarity loss term and the projection matrix regularization loss term; The initial projection matrix is ​​updated based on the projection loss. The process then returns to the step of obtaining a single image-text pair sample from multiple image-text pair samples within the query domain to obtain the next image-text pair sample and perform the projection matrix update operation until the obtained projection matrix satisfies the preset convergence condition.

5. The method according to claim 1, characterized in that, In response to receiving a query request, the intent of the query request is identified to determine the intent level and query domain of the query request, including: In response to receiving a query request, a preprocessing task is constructed based on the query request, and the preprocessing task is executed through the FPGA to realize intent recognition of the query request and determine the intent level and query domain of the query request; The method further includes: Based on the intent level and query domain of the query request, a core computing task is constructed and executed by the GPU to achieve activation and masking of the text encoder and visual encoder, acquisition of knowledge graph, encoding and fusion of text vector and image vector, and searching of query results. The process of executing the core computing task through the GPU includes: If there are remaining tokens in the pre-created token bucket, one token is consumed to place the core computing task corresponding to the query request into the request buffer waiting queue, and resource scheduling is performed based on the order of tasks in the request buffer waiting queue. The tokens in the token bucket are generated according to a dynamic token generation rate with a preset token bucket capacity as the limit. The dynamic token generation rate is determined based on the current device hardware health value, the queue depth of the request buffer queue, and the current system load.

6. The method according to claim 5, characterized in that, Resource scheduling is performed based on the order of tasks in the request buffer waiting queue, including: Resource scheduling is performed on each core computing task sequentially according to the order of tasks in the request buffer waiting queue. The resource scheduling for any core computing task is performed in the following manner: Based on the number of activation layers of the text encoder corresponding to the core computing task, the number of activation layers of the corresponding visual encoder, and the number of open channels, the estimated activation computational load of the core computing task is determined. Based on the estimated activation computational load and the computing power of a single GPU instance, the minimum number of GPU instances required for the core computing task is determined. Resource scheduling is performed for the core computing tasks according to the minimum required number of GPU instances.

7. The method according to claim 1, characterized in that, The depth configuration corresponding to the intent level includes a layer range; the text encoder and visual encoder are activated according to the depth configuration, including: Obtain the current device temperature value and the system's effective throughput; Based on the intent entropy value corresponding to the intent level, the current device temperature value, and the system effective throughput, the target activation layer is determined in the layer range; The text encoder and visual encoder are activated at the corresponding layers according to the depth configuration.

8. A dynamic deep multimodal search device based on user intent awareness, characterized in that, The device includes: An intent recognition module is used to respond to a received query request by performing intent recognition on the query request and determining the intent level and query domain of the query request. The depth masking module is used to obtain the depth configuration and global sparsity threshold corresponding to the intent level, activate the corresponding layers of the text encoder and visual encoder according to the depth configuration, and perform structured sparse masking on the channels of the visual encoder whose channel importance is lower than the global sparsity threshold. The text enhancement module is used to obtain a knowledge graph that matches the query request from the domain knowledge graph library corresponding to the query domain, convert the knowledge graph into natural language description text as the enhanced text of the query request, and concatenate the enhanced text after the query text of the query request to obtain the query enhanced text; The result search module is used to encode the query enhanced text into a text vector using the text encoder, encode the query image in the query request into an image vector using the visual encoder, fuse the text vector and the image vector to obtain a query vector, and perform query result search based on the query vector.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images