Context embedding method and device based on recursive feedback enhancement, equipment and medium

Abstract

The invention relates to the technical field of natural language processing, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a context embedding method, device, equipment and medium based on recursive feedback enhancement. And generating joint embedding representation through dimension alignment, splicing and a cross attention mechanism. The system judges whether the input is the first time or not, when the input is the first time, an initial retrieval result of image-text combination is generated based on joint embedding, feedback data is collected and marked as historical data, an initial processing link is returned to form an optimized closed loop, and when the input is not the first time, the historical feedback is utilized to adjust the gating weight of the bidirectional recurrent neural network; and through non-local attention enhancement joint feature semantic association, an improved retrieval result fusing user preferences is finally output by the adjusted network, and dynamic improvement of retrieval precision in multi-round interaction is realized. And the retrieval accuracy based on context embedding is improved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a context embedding method, apparatus, device and medium based on recursive feedback enhancement. Background Art

[0002] With the rapid development of FinTech (financial technology), financial services are becoming increasingly digital and intelligent. Users in financial scenarios often rely on multimodal data, such as text and images, to obtain information, complete transactions, or seek services. However, existing FinTech approaches face numerous challenges in processing this cross-modal data. Existing methods struggle to establish deep semantic connections between text and image features. They are unable to direct text features to key areas of financial information within an image, such as the amount on a bill or the signature clauses in a contract. They also fail to direct image features to key descriptions within the text. Consequently, joint embeddings fail to accurately represent users' true needs. Furthermore, when processing multiple user inputs, they lack effective utilization of user feedback data and are unable to dynamically adjust models based on user feedback on search results. This results in FinTech systems failing to accurately capture user preferences across multiple rounds of interactions with users, leading to discrepancies between the provided financial services and search results, impacting user experience and the efficiency of financial services.

[0003] In the healthcare sector, with the in-depth application of medical information technology and artificial intelligence technologies, cross-modal data processing is playing an increasingly important role in disease diagnosis, treatment plan development, and health management. Doctors and patients often need to combine symptom descriptions (such as "cough, fever, fatigue") with medical image data (such as CT images, MRI images, and pathological slides) to analyze and diagnose their condition. However, cross-modal data processing in this field currently faces significant shortcomings. Existing technical approaches struggle to effectively interact and fuse medical text features with medical image features. For example, a feature described as "lung shadow" in text cannot be accurately mapped to a specific lesion area in a CT image, and tumor boundary features in a medical image cannot establish a close semantic association with descriptions such as "possible malignancy" in text. As a result, the joint embedding representation cannot fully and accurately represent the comprehensive information of the condition. When processing multiple patient consultations and feedback, traditional methods fail to fully utilize patient feedback on diagnosis results and treatment recommendations. The model cannot be dynamically adjusted based on this feedback data, especially when targeting different disease areas. It is impossible to select knowledge experts who are more suitable for the current disease area by reasonably adjusting the model's gating weights. As a result, the medical and health system cannot perform accurate disease analysis and diagnosis based on the patient's specific situation and historical feedback during multiple rounds of interaction, affecting the quality and efficiency of medical services. Summary of the Invention

[0004] The present invention provides a context embedding method, apparatus, computer equipment and medium based on recursive feedback enhancement to solve the problem of low efficiency and low security of identity card verification in various CRS cash machines currently on the market.

[0005] In a first aspect, a context embedding method based on recursive feedback enhancement is provided, comprising:

[0006] Acquiring query text and image data input by a user, and performing preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features;

[0007] Performing feature combination on the preliminary text features and the preliminary image features to obtain a joint embedding representation;

[0008] Determining whether the query text and the image data are input by the user for the first time;

[0009] If it is the first input, outputting an initial search result according to the joint embedding representation, obtaining user feedback data on the initial search result, marking the feedback data as the previous round of feedback data, returning to the query text and image data obtained from the user input, and performing preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features;

[0010] If it is not the first input, the user's previous round of feedback data is obtained, and the gate weight of the preset bidirectional recurrent neural network is adjusted according to the previous round of feedback data to obtain an adjusted bidirectional recurrent neural network;

[0011] Performing non-local feature enhancement on the joint embedding representation according to the previous round of feedback data to obtain enhanced features;

[0012] The adjusted bidirectional recurrent neural network is used to improve retrieval results based on the enhanced feature output.

[0013] In a second aspect, a context embedding device based on recursive feedback enhancement is provided, comprising:

[0014] A feature extraction module is used to obtain the query text and image data input by the user, and perform preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features;

[0015] a feature combination module, configured to combine the preliminary text features and the preliminary image features to obtain a combined embedding representation;

[0016] an input judgment module, configured to judge whether the query text and the image data are input by the user for the first time; if so, outputting an initial search result based on the joint embedding representation, obtaining user feedback data on the initial search result, marking the feedback data as the previous round of feedback data, and returning the result to the feature combination module, wherein the query text and image data input by the user are obtained, and preliminary feature extraction is performed on the query text and the image data to obtain preliminary text features and preliminary image features;

[0017] A weight adjustment module is used to obtain the user's previous round of feedback data, and adjust the gate weights of the preset bidirectional recurrent neural network according to the previous round of feedback data to obtain an adjusted bidirectional recurrent neural network;

[0018] An enhanced retrieval module is used to perform non-local feature enhancement on the joint embedding representation based on the feedback data of the previous round to obtain enhanced features, and use the adjusted bidirectional recurrent neural network to output improved retrieval results based on the enhanced features.

[0019] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned context embedding method based on recursive feedback enhancement are implemented.

[0020] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned context embedding method based on recursive feedback enhancement are implemented.

[0021] In the solution implemented by the above-mentioned context embedding method, device, computer equipment and storage medium based on recursive feedback enhancement, the query text and image data input by the user can be obtained, and preliminary feature extraction can be performed on the query text and the image data to obtain preliminary text features and preliminary image features, and the preliminary text features and the preliminary image features can be combined to obtain a joint embedding representation, and it can be determined whether the query text and the image data are the first input of the user. If they are the first input, the initial search result is output according to the joint embedding representation, and the user's feedback data on the initial search result is obtained. After marking the feedback data as the previous round of feedback data, the result is returned. The step of obtaining the query text and image data input by the user, and performing preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features, if it is not the first input, obtaining the user's previous round of feedback data, adjusting the gate weights of the preset bidirectional recurrent neural network according to the previous round of feedback data to obtain an adjusted bidirectional recurrent neural network, performing non-local feature enhancement on the joint embedding representation according to the previous round of feedback data to obtain enhanced features, and using the adjusted bidirectional recurrent neural network to improve the retrieval results according to the enhanced feature output, thereby improving the accuracy of retrieval based on context embedding and improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Figure 1 1 is a schematic diagram of an application environment of a context embedding method based on recursive feedback enhancement in one embodiment of the present invention;

[0024] Figure 2 1 is a flow chart of a context embedding method based on recursive feedback enhancement in one embodiment of the present invention;

[0025] Figure 3 1 is a structural diagram of a context embedding device based on recursive feedback enhancement in one embodiment of the present invention;

[0026] Figure 4 is a structural diagram of a computer device in one embodiment of the present invention;

[0027] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0029] The context embedding method based on recursive feedback enhancement provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, a client communicates with a server through a network. The server can obtain the query text and image data input by the user through the client, and perform preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features, perform feature combination on the preliminary text features and the preliminary image features to obtain a joint embedding representation, determine whether the query text and the image data are the user's first input, and if so, output the initial retrieval result according to the joint embedding representation, obtain the user's feedback data on the initial retrieval result, mark the feedback data as the previous round of feedback data, and then return to the step of obtaining the query text and image data input by the user, and perform preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features. If it is not the first input, obtain the user's previous round of feedback data, adjust the gate weights of the preset bidirectional recurrent neural network according to the previous round of feedback data, and obtain an adjusted bidirectional recurrent neural network. Perform non-local feature enhancement on the joint embedding representation according to the previous round of feedback data to obtain enhanced features, and use the adjusted bidirectional recurrent neural network to output improved retrieval results according to the enhanced features, thereby improving retrieval efficiency and accuracy. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablet computers, and portable wearable devices. The server can be implemented as an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0030] See also Figure 2 As shown, Figure 2 A flowchart of a context embedding method based on recursive feedback enhancement provided by an embodiment of the present invention includes the following steps:

[0031] S1. Obtaining query text and image data input by a user, and performing preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features.

[0032] In the field of FinTech, the query text may include transaction records (such as transfer amounts and frequency), account information (such as balances and transaction records), loan application text (such as proof of income and credit history), customer service inquiries (such as questions about financial products), and financial news keywords entered by users into financial inquiry systems. The image data may include ID / bank card photos (for identity verification), images of bills (such as invoices and checks), screenshots of securities K-line charts, company logo images (for brand recognition), and handwritten signature images (for electronic contract confirmation).

[0033] In the medical technology field, the query text can be patient medical record descriptions (such as symptoms and medical history) entered by users in a medical query system, diagnostic report text (such as blood test results), prescription content, medical literature keywords, diet / exercise records entered through a health management app, etc. The image data can be medical images (such as X-rays, CT / MRI scans), pathological section images, photos of skin lesions, medical equipment images (such as ultrasound instrument interfaces), and pictures of drug packaging boxes (for medication consultation).

[0034] In an embodiment of the present invention, a pre-trained cross-modal encoder may be used to perform preliminary feature extraction on the query text and the image data.

[0035] In detail, the cross-modal encoder can simultaneously process pre-trained models of two different modal data (such as BERT and RoBERTa) of text and image.

[0036] In the embodiment of the present invention, the performing of preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features includes:

[0037] Performing word segmentation processing on the query text to obtain a word segmentation sequence;

[0038] Performing denoising processing on the word segmentation sequence to obtain a denoised word segmentation sequence;

[0039] Generate word embeddings of the denoised word segmentation sequence using a multi-layer Transformer structure with a preset encoder to obtain preliminary text features;

[0040] Performing size standardization processing on the image data to obtain a standard size image;

[0041] Normalizing the pixel value of each pixel of the standard size image to obtain a normalized image;

[0042] The normalized image is convolved using the convolution layer of the encoder to obtain preliminary image features.

[0043] In detail, the word segmentation process is to split the input query text (such as "red flowers") into smaller semantic units (such as "red", "of", "flowers") to facilitate word-by-word processing by the model.

[0044] In detail, the denoising process is to filter out redundant information (such as repeated words, meaningless modal particles) or correct errors (such as spelling errors) in word segmentation to improve the quality of subsequent features.

[0045] In detail, the word embedding converts the segmented text into a continuous numerical vector (e.g., each word corresponds to a 128-dimensional vector), so that the model can understand the semantics mathematically.

[0046] In detail, the size normalization processing performed on the image data is to adjust the image to a fixed size (such as 224*224).

[0047] In detail, the normalization process is to scale the pixel value of each pixel of the standard size image to the range of 0-1 to avoid instability in model training caused by excessively large values.

[0048] In an embodiment of the present invention, by obtaining the query text and image data input by the user and performing preliminary feature extraction on the query text and the image data, preliminary text features and preliminary image features are obtained, which can improve the efficiency of subsequent feature combination of the preliminary text features and the preliminary image features.

[0049] S2. Perform feature combination on the preliminary text features and the preliminary image features to obtain a joint embedding representation.

[0050] In the embodiment of the present invention, the preliminary text features and the preliminary image features are combined to obtain a joint embedding representation, which can be generated by a cross-modal interaction layer (such as the cross attention of Transformer).

[0051] In the embodiment of the present invention, the step of combining the preliminary text features and the preliminary image features to obtain a joint embedding representation includes:

[0052] Dimensionally aligning the preliminary text features and the preliminary image features to obtain aligned text features and aligned image features;

[0053] Splicing the aligned text features and the aligned image features to obtain a spliced ​​feature;

[0054] Using the cross-attention mechanism of the Transformer architecture to model the interactive relationship between the preliminary text features and the preliminary image features;

[0055] A preset nonlinear activation function is used to perform nonlinear transformation on the splicing features according to the interaction relationship to obtain a joint embedding representation.

[0056] In the embodiment of the present invention, the dimensions of the preliminary text features and the preliminary image features are aligned, and their dimensions are unified (eg, both are adjusted to 512 dimensions) through linear transformation to ensure that subsequent splicing is feasible.

[0057] In detail, the cross-attention mechanism of the Transformer architecture is used to model the interactive relationship between the preliminary text features and the preliminary image features. Through the cross-attention layer of the Transformer, the text features are made to "focus on" the key areas in the image (such as the text "red" corresponds to the red petals in the image), while the image features "focus on" the key descriptions in the text, thereby strengthening the semantic association between the two.

[0058] In the embodiment of the present invention, by performing feature combination on the preliminary text features and the preliminary image features to obtain a joint embedding representation, the efficiency of subsequent generation of enhanced features can be improved.

[0059] S3. Determine whether the query text and the image data are input by the user for the first time.

[0060] In an embodiment of the present invention, the determination of whether the query text and the image data are the user's first input is performed by determining whether the query text and the image data have a previous context. If there is no previous context, it is confirmed as the first input; if there is a previous context, it is confirmed as not the first input.

[0061] In an embodiment of the present invention, determining whether the query text and the image data are input by the user for the first time includes:

[0062] When receiving the query text and the image data input by the user, obtaining the user ID of the user;

[0063] Obtaining user context cache data from a preset database table according to the user ID;

[0064] Determining whether the user's current input has a previous context based on the context cache data;

[0065] If there is no such context, it is determined that the query text and the image data are input by the user for the first time;

[0066] If so, it is determined that the query text and the image data are not input by the user for the first time.

[0067] If it is the first input, execute S4, output the initial retrieval result according to the joint embedding representation, obtain the user's feedback data on the initial retrieval result, mark the feedback data as the previous round of feedback data, return to the query text and image data input by the user, and perform preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features.

[0068] In an embodiment of the present invention, the feedback data includes the user's displayed score and implicit behavior data. The displayed score can be the user's satisfaction score for the retrieval results, and the implicit behavior data can be the user's operational behavior data on the retrieval results (such as clicking on an image or text field in the retrieval results).

[0069] In the embodiment of the present invention, outputting the initial search results according to the joint embedding representation includes:

[0070] Performing dimension adjustment on the joint embedding representation to obtain a dimension adjusted embedding representation;

[0071] Adjusting the embedding representation according to the dimension to retrieve the text fragment in a preset knowledge base;

[0072] Generate a simplified schematic diagram using a visual generative model to adjust the image features in the embedded representation according to the dimension;

[0073] Retrieving a matching image from a preset image library according to the simplified schematic diagram;

[0074] The text segment and the matching image are dimensionally aligned and then outputted in a preset format to obtain an initial retrieval result.

[0075] In detail, the text segment and the matching image are dimensionally aligned and then outputted in a preset format, and the text segment and the matching image are combined in a manner such as left image and right text to form an initial search result.

[0076] If it is not the first input, execute S5, obtain the user's previous round of feedback data, and adjust the gate weight of the preset bidirectional recurrent neural network according to the previous round of feedback data to obtain an adjusted bidirectional recurrent neural network.

[0077] In an embodiment of the present invention, the bidirectional recurrent neural network (BiRNN) is a recurrent network that can simultaneously capture past and future context information.

[0078] In an embodiment of the present invention, the gate weight adjustment of the preset bidirectional recurrent neural network based on the previous round of feedback data can be performed by adopting a mixture of experts (MoE) architecture, which includes k independent feedforward networks (Experts) and dynamically selects the expert combination through gate weights.

[0079] In detail, the hybrid expert architecture includes multiple independent feedforward networks (experts), which dynamically selects the expert combination that best suits the current task through gating weights, and selects experts based on the field of the question asked by the user.

[0080] In the field of financial technology, the hybrid expert architecture may include financial knowledge experts, insurance knowledge experts, legal provisions experts, and claims terms experts, etc.

[0081] In the field of medical health, the hybrid expert architecture may include surgical knowledge experts, internal medicine knowledge experts, orthopedic knowledge experts, and tumor knowledge experts, etc.

[0082] In an embodiment of the present invention, the gate weight adjustment of the preset bidirectional recurrent neural network is performed according to the previous round of feedback data to obtain the adjusted bidirectional recurrent neural network, including:

[0083] Converting the previous round of feedback data into a numerical signal to obtain a feedback numerical signal;

[0084] Extracting the timing characteristics of the feedback numerical signal;

[0085] Mapping the temporal features into input dimensions of a hybrid expert network in the bidirectional recurrent neural network using a learnable projection matrix to obtain a dimension mapping signal;

[0086] generating a controller signal using the hybrid expert network according to preset gating weights in the bidirectional recurrent neural network and the joint embedding representation;

[0087] generating a gating signal based on the controller signal and the joint embedding representation using a linear layer and a sigmoid activation function;

[0088] A reinforcement learning algorithm is used to update the gating weights of the bidirectional recurrent neural network according to the gating signal and the dimensional mapping signal to obtain an adjusted bidirectional recurrent neural network.

[0089] In an embodiment of the present invention, converting the previous round of feedback data into a numerical signal may refer to converting the displayed score in the previous round of feedback data into a numerical value, such as converting a score of 5 into a numerical value of 5, and converting the implicit behavior in the previous round of feedback data (such as clicking on the third picture and staying for 15 seconds) into a numerical value, such as [click position 3, stay time 15].

[0090] In the embodiment of the present invention, the extraction of the time series features of the feedback value signal is to capture the time sequence and trend of user feedback (such as the gradual increase / decrease of the feedback scores in the past three times) to provide time series information for the dynamic adjustment model.

[0091] In an embodiment of the present invention, the use of a learnable projection matrix to map the time series features to the input dimension of the hybrid expert network in the bidirectional recurrent neural network is to map the time series features from D1 dimension to D2 dimension (such as from 64 dimensions to 128 dimensions) through matrix multiplication, thereby solving the problem of mismatch between the time series features and the input dimension of the MoE network.

[0092] In an embodiment of the present invention, the use of a linear layer and a Sigmoid activation function to generate a gating signal according to the controller signal and the joint embedding representation is to use a linear layer and a Sigmoid activation function to generate a gating signal according to the controller signal and the joint embedding representation, and then compress the output value to the [0,1] interval through the Sigmoid activation function (e.g., outputting 0.8 indicates "strong gating activation" and 0.2 indicates "weak gating activation").

[0093] In an embodiment of the present invention, the use of a reinforcement learning algorithm to update the gating weights of the bidirectional recurrent neural network according to the gating signal and the dimension mapping signal is a method in which the reinforcement learning algorithm (such as PPO, DQN) uses "user satisfaction" as a reward signal (such as a reward of +1 when the user clicks / collects a result, and a reward of -1 when the user skips / gives a bad review). The algorithm calculates the update amount of the gating weights (such as increasing the weight of experts who are good at processing "high-scoring feedback" and reducing the weight of experts who process "low-scoring feedback") based on the gating signal (current adjustment direction) and the dimension mapping signal (feedback mode). The updated gating weights act directly on the bidirectional recurrent neural network, allowing it to more accurately capture user preferences in subsequent rounds of interaction.

[0094] S6. Perform non-local feature enhancement on the joint embedding representation according to the previous round of feedback data to obtain enhanced features.

[0095] In the embodiment of the present invention, the non-local feature enhancement of the joint embedding representation based on the previous round of feedback data is to strengthen the expression of key information in the joint embedding by capturing the long-distance dependency between features.

[0096] In the embodiment of the present invention, the step of performing non-local feature enhancement on the joint embedding representation according to the previous round of feedback data to obtain enhanced features includes:

[0097] constructing a local neighborhood of the joint embedding representation;

[0098] Determining, according to the local neighborhood, a feature vector belonging to the local neighborhood in the joint embedding representation to obtain a neighborhood feature vector set;

[0099] Calculating the attention weight between each feature vector in the neighborhood feature vector set, and constructing an attention matrix according to the attention weight;

[0100] Performing a weighted summation of all feature vectors in the joint embedding representation according to the attention matrix to obtain an enhanced representation;

[0101] Transforming the enhanced representation using a learnable weight matrix to obtain a dimensionally transformed representation;

[0102] fusing the dimensionally transformed representation with the joint embedded representation using a residual connection to obtain a fused representation;

[0103] The fused representation is layer-normalized to obtain enhanced features.

[0104] In detail, the construction of the local neighborhood of the joint embedding representation is to divide the feature vectors of the joint embedding into multiple local groups (such as 10 vectors per group), and each group is regarded as a "neighborhood".

[0105] In detail, the method of determining the feature vector belonging to the local neighborhood in the joint embedding representation based on the local neighborhood is to filter out vectors related to the current feature semantics from each neighborhood (such as the vector of the "flower opening state" in the image corresponding to the text "flowering period").

[0106] In detail, calculating the attention weight between each feature vector in the neighborhood feature vector set is to calculate the cosine similarity between each feature vector in the neighborhood feature vector set.

[0107] In detail, the use of a learnable weight matrix to perform dimensional transformation on the enhanced representation to obtain a dimensional transformed representation is to adjust the dimension of the enhanced representation to make it compatible with subsequent operations (such as residual connections), and to introduce learnable parameters to enhance the model's expressive power. The enhanced representation can be linearly transformed by matrix multiplication.

[0108] In detail, the learnable weight matrix is ​​a parameter matrix of shape [D, D'] (D' is the target dimension, which is usually the same as the dimension D of the original joint embedding, i.e., D'=D).

[0109] In detail, the residual connection is used to fuse the dimension transformation representation with the joint embedding representation to obtain a fused representation in order to retain the original joint embedding information and avoid losing key features during the enhancement process.

[0110] S7. Utilize the adjusted bidirectional recurrent neural network to output improved retrieval results based on the enhanced features.

[0111] In an embodiment of the present invention, the step of using the adjusted bidirectional recurrent neural network to output an improved retrieval result based on the enhanced feature is the same as the step of outputting the initial retrieval result based on the joint embedding representation, and will not be repeated here.

[0112] As can be seen in the above solution, a pre-trained cross-modal encoder performs preliminary feature extraction on the user's query text and image data. After word segmentation and denoising, the text is passed through a Transformer to generate word embeddings to obtain preliminary text features. The image is normalized in size and pixels, and then passed through a convolutional layer to obtain preliminary image features. Next, the preliminary text and image features are dimensionally aligned and concatenated, and the interactive relationship is modeled using the Transformer's cross-attention mechanism. A joint embedding representation is generated through nonlinear transformations. The system then determines whether the user input is the first time. If so, the joint embedding dimension is adjusted. The knowledge base searches for text snippets, a simplified image diagram is generated using a visual generative model, and matching images are retrieved. Initial search results are output in a preset format. User explicit ratings and implicit behavioral feedback are collected, marked as the previous round of feedback, and the system returns to the feature extraction step. If so, the previous round of feedback is obtained, converted into numerical signals, and temporal features are extracted. These signals are then mapped to the input dimension of the hybrid expert network using a projection matrix. The controller and gating signals are then generated by combining the joint embedding. The gating weights of the bidirectional recurrent neural network are updated using a reinforcement learning algorithm. Non-local feature enhancement is performed on the joint embedding based on the feedback data. Enhanced features are obtained by constructing local neighborhoods, calculating attention weights, performing weighted summation, dimensionality transformation, residual connections, and layer normalization. Finally, the adjusted bidirectional recurrent neural network is combined with the enhanced feature output to improve the search results. The result generation logic is consistent with the initial search. Dynamic optimization improves search accuracy over multiple rounds of interaction, making it suitable for accurate response scenarios involving mixed image and text queries.

[0113] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0114] In one embodiment, a context embedding device based on recursive feedback enhancement is provided, and the context embedding device based on recursive feedback enhancement corresponds one-to-one to the context embedding method based on recursive feedback enhancement in the above embodiment. Figure 3 As shown, the context embedding device based on recursive feedback enhancement includes a feature extraction module 101, a feature combination module 102, an input judgment module 103, a weight adjustment module 104, and an enhanced retrieval module 105. The functional modules are described in detail as follows:

[0115] The feature extraction module 101 is used to obtain the query text and image data input by the user, and perform preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features;

[0116] A feature combining module 102 is configured to combine the preliminary text features and the preliminary image features to obtain a combined embedding representation;

[0117] An input determination module 103 is configured to determine whether the query text and the image data are input by the user for the first time. If so, the module outputs an initial search result based on the joint embedding representation, obtains user feedback data on the initial search result, marks the feedback data as the previous round of feedback data, and then returns to the step of obtaining the query text and image data input by the user, performing preliminary feature extraction on the query text and the image data, and obtaining preliminary text features and preliminary image features in the feature combination module 102.

[0118] The weight adjustment module 104 is used to obtain the user's previous round of feedback data, and adjust the gate weights of the preset bidirectional recurrent neural network according to the previous round of feedback data to obtain an adjusted bidirectional recurrent neural network;

[0119] The enhanced retrieval module 105 is used to perform non-local feature enhancement on the joint embedding representation based on the previous round of feedback data to obtain enhanced features, and use the adjusted bidirectional recurrent neural network to output improved retrieval results based on the enhanced features.

[0120] In one embodiment, the feature extraction module 101, when performing the preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features, is specifically configured to:

[0121] Performing word segmentation processing on the query text to obtain a word segmentation sequence;

[0122] Performing denoising processing on the word segmentation sequence to obtain a denoised word segmentation sequence;

[0123] Generate word embeddings of the denoised word segmentation sequence using a multi-layer Transformer structure with a preset encoder to obtain preliminary text features;

[0124] Performing size standardization processing on the image data to obtain a standard size image;

[0125] Normalizing the pixel value of each pixel of the standard size image to obtain a normalized image;

[0126] The normalized image is convolved using the convolution layer of the encoder to obtain preliminary image features.

[0127] In one embodiment, the feature combining module 102, when performing the feature combining of the preliminary text features and the preliminary image features to obtain a joint embedding representation, is specifically configured to:

[0128] Dimensionally aligning the preliminary text features and the preliminary image features to obtain aligned text features and aligned image features;

[0129] Splicing the aligned text features and the aligned image features to obtain a spliced ​​feature;

[0130] Using the cross-attention mechanism of the Transformer architecture to model the interactive relationship between the preliminary text features and the preliminary image features;

[0131] A preset nonlinear activation function is used to perform nonlinear transformation on the splicing features according to the interaction relationship to obtain a joint embedding representation.

[0132] In one embodiment, the input determination module 103, when determining whether the query text and the image data are input by the user for the first time, is specifically configured to:

[0133] When receiving the query text and the image data input by the user, obtaining the user ID of the user;

[0134] Obtaining user context cache data from a preset database table according to the user ID;

[0135] Determining whether the user's current input has a previous context based on the context cache data;

[0136] If there is no such context, it is determined that the query text and the image data are input by the user for the first time;

[0137] If so, it is determined that the query text and the image data are not input by the user for the first time.

[0138] In one embodiment, the input judgment module 103, when outputting the initial search result according to the joint embedding representation, is specifically configured to:

[0139] Performing dimension adjustment on the joint embedding representation to obtain a dimension adjusted embedding representation;

[0140] Adjusting the embedding representation according to the dimension to retrieve the text fragment in a preset knowledge base;

[0141] Generate a simplified schematic diagram using a visual generative model to adjust the image features in the embedded representation according to the dimension;

[0142] Retrieving a matching image from a preset image library according to the simplified schematic diagram;

[0143] The text segment and the matching image are dimensionally aligned and then outputted in a preset format to obtain an initial retrieval result.

[0144] In one embodiment, the weight adjustment module 104, when performing the gated weight adjustment of the preset bidirectional recurrent neural network according to the previous round of feedback data to obtain the adjusted bidirectional recurrent neural network, is specifically configured to:

[0145] Converting the previous round of feedback data into a numerical signal to obtain a feedback numerical signal;

[0146] Extracting the timing characteristics of the feedback numerical signal;

[0147] Mapping the temporal features into input dimensions of a hybrid expert network in the bidirectional recurrent neural network using a learnable projection matrix to obtain a dimension mapping signal;

[0148] generating a controller signal using the hybrid expert network according to preset gating weights in the bidirectional recurrent neural network and the joint embedding representation;

[0149] generating a gating signal based on the controller signal and the joint embedding representation using a linear layer and a sigmoid activation function;

[0150] A reinforcement learning algorithm is used to update the gating weights of the bidirectional recurrent neural network according to the gating signal and the dimensional mapping signal to obtain an adjusted bidirectional recurrent neural network.

[0151] In one embodiment, the enhanced retrieval module 105, when performing the non-local feature enhancement on the joint embedding representation according to the previous round of feedback data to obtain the enhanced features, is specifically configured to:

[0152] constructing a local neighborhood of the joint embedding representation;

[0153] Determining, according to the local neighborhood, a feature vector belonging to the local neighborhood in the joint embedding representation to obtain a neighborhood feature vector set;

[0154] Calculating the attention weight between each feature vector in the neighborhood feature vector set, and constructing an attention matrix according to the attention weight;

[0155] Performing a weighted summation of all feature vectors in the joint embedding representation according to the attention matrix to obtain an enhanced representation;

[0156] Transforming the enhanced representation using a learnable weight matrix to obtain a dimensionally transformed representation;

[0157] fusing the dimensionally transformed representation with the joint embedded representation using a residual connection to obtain a fused representation;

[0158] The fused representation is layer-normalized to obtain enhanced features.

[0159] This paper provides a contextual embedding device based on recursive feedback enhancement. A pre-trained cross-modal encoder extracts preliminary features from user-entered query text and image data. After word segmentation and denoising, the text is passed through a Transformer to generate word embeddings to obtain preliminary text features. The image is then dimensionally and pixel-normalized and passed through a convolutional layer to obtain preliminary image features. The preliminary text and image features are then dimensionally aligned and concatenated, and their interactions are modeled using the Transformer's cross-attention mechanism. A joint embedding representation is generated through nonlinear transformations. The system then determines whether the user input is the first time. If so, the joint embedding dimension is adjusted. The knowledge base searches for text snippets, a simplified image diagram is generated using a visual generative model, and matching images are retrieved. Initial search results are output in a preset format. User explicit ratings and implicit behavioral feedback are collected, marked as the previous round of feedback, and the system returns to the feature extraction step. If so, the previous round of feedback is obtained, converted into numerical signals, and temporal features are extracted. These signals are then mapped to the input dimension of the hybrid expert network using a projection matrix. The controller and gating signals are then generated by combining the joint embedding. The gating weights of the bidirectional recurrent neural network are updated using a reinforcement learning algorithm. Non-local feature enhancement is performed on the joint embedding based on the feedback data. Enhanced features are obtained by constructing local neighborhoods, calculating attention weights, performing weighted summation, dimensionality transformation, residual connections, and layer normalization. Finally, the adjusted bidirectional recurrent neural network is combined with the enhanced feature output to improve the search results. The result generation logic is consistent with the initial search. Dynamic optimization improves search accuracy over multiple rounds of interaction, making it suitable for accurate response scenarios involving mixed image and text queries.

[0160] Regarding the specific limitations of the context embedding device based on recursive feedback enhancement, please refer to the limitations of the context embedding method based on recursive feedback enhancement above, which will not be repeated here. The various modules in the above-mentioned context embedding device based on recursive feedback enhancement can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0161] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a context embedding method based on recursive feedback enhancement.

[0162] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the client side of a context embedding method based on recursive feedback enhancement

[0163] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0164] Acquiring query text and image data input by a user, and performing preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features;

[0165] Performing feature combination on the preliminary text features and the preliminary image features to obtain a joint embedding representation;

[0166] Determining whether the query text and the image data are input by the user for the first time;

[0167] If it is the first input, outputting an initial search result according to the joint embedding representation, obtaining user feedback data on the initial search result, marking the feedback data as the previous round of feedback data, returning to the query text and image data obtained from the user input, and performing preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features;

[0168] If it is not the first input, the user's previous round of feedback data is obtained, and the gate weight of the preset bidirectional recurrent neural network is adjusted according to the previous round of feedback data to obtain an adjusted bidirectional recurrent neural network;

[0169] Performing non-local feature enhancement on the joint embedding representation according to the previous round of feedback data to obtain enhanced features;

[0170] The adjusted bidirectional recurrent neural network is used to improve retrieval results based on the enhanced feature output.

[0171] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0172] Acquiring query text and image data input by a user, and performing preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features;

[0173] Performing feature combination on the preliminary text features and the preliminary image features to obtain a joint embedding representation;

[0174] Determining whether the query text and the image data are input by the user for the first time;

[0175] If it is the first input, outputting an initial search result according to the joint embedding representation, obtaining user feedback data on the initial search result, marking the feedback data as the previous round of feedback data, returning to the query text and image data obtained from the user input, and performing preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features;

[0176] If it is not the first input, the user's previous round of feedback data is obtained, and the gate weight of the preset bidirectional recurrent neural network is adjusted according to the previous round of feedback data to obtain an adjusted bidirectional recurrent neural network;

[0177] Performing non-local feature enhancement on the joint embedding representation according to the previous round of feedback data to obtain enhanced features;

[0178] The adjusted bidirectional recurrent neural network is used to improve retrieval results based on the enhanced feature output.

[0179] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0180] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0181] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0182] Finally, it should be noted that if software tools or components other than those of our company appear in the application examples, they are merely for illustration and do not represent actual use. The above-described embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above-described embodiments or replace some of the technical features therein with equivalents. Such modifications or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention and should be included within the scope of protection of the present invention.

Claims

1. A contextual embedding method based on recursive feedback enhancement, characterized in that: include: Acquiring query text and image data input by a user, and performing preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features; Performing feature combination on the preliminary text features and the preliminary image features to obtain a joint embedding representation; Determining whether the query text and the image data are input by the user for the first time; If it is the first input, outputting an initial search result according to the joint embedding representation, obtaining user feedback data on the initial search result, marking the feedback data as the previous round of feedback data, returning to the query text and image data obtained from the user input, and performing preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features; If it is not the first input, the user's previous round of feedback data is obtained, and the gate weight of the preset bidirectional recurrent neural network is adjusted according to the previous round of feedback data to obtain an adjusted bidirectional recurrent neural network; Performing non-local feature enhancement on the joint embedding representation according to the previous round of feedback data to obtain enhanced features; The adjusted bidirectional recurrent neural network is used to improve retrieval results based on the enhanced feature output.

2. The context embedding method based on recursive feedback enhancement according to claim 1, characterized in that The performing preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features includes: Performing word segmentation processing on the query text to obtain a word segmentation sequence; Performing denoising processing on the word segmentation sequence to obtain a denoised word segmentation sequence; Generate word embeddings of the denoised word segmentation sequence using a multi-layer Transformer structure with a preset encoder to obtain preliminary text features; Performing size standardization processing on the image data to obtain a standard size image; Normalizing the pixel value of each pixel of the standard size image to obtain a normalized image; The normalized image is convolved using the convolution layer of the encoder to obtain preliminary image features.

3. The context embedding method based on recursive feedback enhancement according to claim 1, characterized in that The combining of the preliminary text features and the preliminary image features to obtain a joint embedding representation includes: Dimensionally aligning the preliminary text features and the preliminary image features to obtain aligned text features and aligned image features; Splicing the aligned text features and the aligned image features to obtain a spliced ​​feature; Using the cross-attention mechanism of the Transformer architecture to model the interactive relationship between the preliminary text features and the preliminary image features; A preset nonlinear activation function is used to perform nonlinear transformation on the splicing features according to the interaction relationship to obtain a joint embedding representation.

4. The context embedding method based on recursive feedback enhancement according to claim 1, wherein: The determining whether the query text and the image data are input by the user for the first time includes: When receiving the query text and the image data input by the user, obtaining the user ID of the user; Obtaining user context cache data from a preset database table according to the user ID; Determining whether the user's current input has a previous context based on the context cache data; If there is no such context, it is determined that the query text and the image data are input by the user for the first time; If so, it is determined that the query text and the image data are not input by the user for the first time.

5. The context embedding method based on recursive feedback enhancement according to claim 1, wherein: Outputting an initial retrieval result according to the joint embedding representation includes: Performing dimension adjustment on the joint embedding representation to obtain a dimension adjusted embedding representation; Adjusting the embedding representation according to the dimension to retrieve the text fragment in a preset knowledge base; Generate a simplified schematic diagram using a visual generative model to adjust the image features in the embedded representation according to the dimension; Retrieving a matching image from a preset image library according to the simplified schematic diagram; The text segment and the matching image are dimensionally aligned and then outputted in a preset format to obtain an initial retrieval result.

6. The context embedding method based on recursive feedback enhancement according to claim 1, wherein: The gate weight adjustment of the preset bidirectional recurrent neural network is performed according to the feedback data of the previous round to obtain an adjusted bidirectional recurrent neural network, including: Converting the previous round of feedback data into a numerical signal to obtain a feedback numerical signal; Extracting the timing characteristics of the feedback numerical signal; Mapping the temporal features into input dimensions of a hybrid expert network in the bidirectional recurrent neural network using a learnable projection matrix to obtain a dimension mapping signal; generating a controller signal using the hybrid expert network according to preset gating weights in the bidirectional recurrent neural network and the joint embedding representation; generating a gating signal based on the controller signal and the joint embedding representation using a linear layer and a sigmoid activation function; A reinforcement learning algorithm is used to update the gating weights of the bidirectional recurrent neural network according to the gating signal and the dimensional mapping signal to obtain an adjusted bidirectional recurrent neural network.

7. The context embedding method based on recursive feedback enhancement according to claim 1, wherein: The performing non-local feature enhancement on the joint embedding representation according to the previous round of feedback data to obtain enhanced features includes: constructing a local neighborhood of the joint embedding representation; Determine, according to the local neighborhood, a feature vector belonging to the local neighborhood in the joint embedding representation, to obtain a neighborhood feature vector set; Calculating the attention weight between each feature vector in the neighborhood feature vector set, and constructing an attention matrix according to the attention weight; Performing a weighted summation of all feature vectors in the joint embedding representation according to the attention matrix to obtain an enhanced representation; Transforming the enhanced representation using a learnable weight matrix to obtain a dimensionally transformed representation; fusing the dimensionally transformed representation with the joint embedded representation using a residual connection to obtain a fused representation; The fused representation is layer-normalized to obtain enhanced features.

8. A context embedding device based on recursive feedback enhancement, characterized in that: include: A feature extraction module is used to obtain the query text and image data input by the user, and perform preliminary feature extraction on the query text and the image data to obtain preliminary text features and preliminary image features; a feature combination module, configured to combine the preliminary text features and the preliminary image features to obtain a combined embedding representation; an input judgment module, configured to judge whether the query text and the image data are input by the user for the first time; if so, outputting an initial search result based on the joint embedding representation, obtaining user feedback data on the initial search result, marking the feedback data as the previous round of feedback data, and returning the result to the feature combination module, wherein the query text and image data input by the user are obtained, and preliminary feature extraction is performed on the query text and the image data to obtain preliminary text features and preliminary image features; A weight adjustment module is used to obtain the user's previous round of feedback data, and adjust the gate weights of the preset bidirectional recurrent neural network according to the previous round of feedback data to obtain an adjusted bidirectional recurrent neural network; An enhanced retrieval module is used to perform non-local feature enhancement on the joint embedding representation based on the feedback data of the previous round to obtain enhanced features, and use the adjusted bidirectional recurrent neural network to output improved retrieval results based on the enhanced features.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the context embedding method based on recursive feedback enhancement are implemented as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the context embedding method based on recursive feedback enhancement are implemented.

Images