Intelligent question-answering methods, devices, computer equipment, and media based on text-image interaction
By employing multimodal content parsing and joint reasoning technologies, the problem of insufficient speed and accuracy in text and image question answering has been solved, enabling efficient and accurate question answering in the fields of healthcare and fintech.
Patent Information
- Application Number
- CN202611422394.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-14
- Publication Date
- 2026-11-17
AI Technical Summary
Existing general text and image question answering solutions cannot effectively improve the speed and accuracy of text and image interactive question answering in vertical industries such as healthcare and fintech, which have high professionalism, high compliance requirements, and high precision constraints.
Employing multimodal content parsing, text Transformer encoding, vector quantization encoder, visual Transformer encoder, and a pre-trained visual language model, standard response results are generated through image encoding compression and noise reduction, bimodal feature dimension unification, visual semantic interference suppression, and multimodal joint accurate reasoning.
It simultaneously improved the response speed and output accuracy of text and image Q&A, reduced interference from invalid information, and improved the accuracy and efficiency of Q&A.
Smart Images

Figure CN123115348A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and specifically to an intelligent question-answering method, apparatus, computer device, and computer-readable storage medium based on graphic and text interaction. Background Technology
[0002] Currently, with the rapid implementation of large-scale AI visual language models, client-side text-based interactive intelligent question-and-answer services are becoming widely adopted in the two major vertical sectors of healthcare and fintech. In healthcare scenarios, users often upload real-life photos of their skin, screenshots of medical images, and photos of test reports, along with text descriptions of symptoms, medication history, and examination requests. In fintech scenarios, users upload financial statement charts, screenshots of investment products, credit reports, and transaction records, combined with text-based inquiries about investment risks, creditworthiness, and interpretation of financial statements. Text-based interactive systems can compensate for the lack of information in purely text-based descriptions, significantly improving the information completeness and scenario adaptability of intelligent question-and-answer systems. Therefore, text-based interactive intelligent question-and-answer systems have become a core tool for the industry's digital transformation.
[0003] Currently, the mainstream general text and image question answering solutions use a large visual language model to directly complete reasoning and response. This architecture has basic interactive capabilities in general text and image chat scenarios, but it cannot guarantee the speed and accuracy of responses when facing vertical industries with high professionalism, high compliance requirements, and high precision constraints, such as healthcare and fintech.
[0004] Therefore, how to provide an intelligent question-answering method, device, computer equipment, and computer-readable storage medium based on graphic and text interaction, which can effectively improve the speed and accuracy of responses during graphic and text interactive question-answering, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide an intelligent question-answering method, device, computer equipment and computer-readable storage medium based on graphic and text interaction, aiming to solve the problem of how to effectively improve the speed and accuracy of responses in graphic and text interactive question-answering.
[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides an intelligent question-answering method based on image-text interaction, comprising: The system obtains query information input by the target user through the client, performs multimodal content parsing on the query information, and extracts the query text and image data associated with the query text from the parsing results. The query text is segmented to obtain text words, and the text words are encoded using a text Transformer encoder to generate a text word feature sequence. The image data is discretized and encoded at a low level using a vector quantization encoder to generate a visual code embedding. The visual code embedding is then input into a multilayer perceptron for dimension adaptation mapping to obtain transformed features. The transformed features are then input into a visual Transformer encoder configured with a causal attention mechanism to generate semantic visual features. The semantic visual features are concatenated with the text lexical feature sequence to form a unified multimodal input sequence. Based on the unified multimodal input sequence, a pre-trained visual language large model is used to perform joint text-image semantic reasoning to generate a standard response result for the query information. The standard response result is then pushed to the client.
[0007] Secondly, the present invention provides an intelligent question-answering device based on graphic-text interaction, comprising: The acquisition module is used to acquire the query information input by the target user through the client, perform multimodal content parsing on the query information, and extract the query text and image data associated with the query text from the parsing results; The first encoding module is used to perform word segmentation on the query text to obtain text words, and to encode the text words using a text Transformer encoder to generate a text word feature sequence. The second encoding module is used to perform low-level discretization encoding on the image data using a vector quantization encoder to generate a visual code embedding. The visual code embedding is then input into a multilayer perceptron for dimension adaptation mapping to obtain transformed features. The transformed features are then input into a visual Transformer encoder configured with a causal attention mechanism to generate semantic visual features. The concatenation module is used to concatenate the semantic visual features with the text lexical feature sequence to form a unified multimodal input sequence, and based on the unified multimodal input sequence, to perform joint text-image semantic reasoning through a pre-trained visual language large model, generate a standard response result for the query information, and push the standard response result to the client.
[0008] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the intelligent question-and-answer method based on graphic-text interaction as described above.
[0009] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the intelligent question-answering method based on graphic-text interaction as described above.
[0010] Compared to existing technologies, this invention provides an intelligent question-answering method, apparatus, computer device, and computer-readable storage medium based on text-image interaction. The method includes: acquiring query information input by a target user through a client; performing multimodal content parsing on the query information; extracting query text and image data associated with the query text from the parsing results; performing word segmentation on the query text to obtain text units, and encoding the text units using a text Transformer encoder to generate a text unit feature sequence; performing low-level discretization encoding on the image data using a vector quantization encoder to generate visual code embeddings; inputting the visual code embeddings into a multilayer perceptron for dimensionality adaptation mapping to obtain transformed features; inputting the transformed features into a visual Transformer encoder configured with a causal attention mechanism to generate semantic visual features; concatenating the semantic visual features with the text unit feature sequence to form a unified multimodal input sequence; and based on the unified multimodal input sequence, performing joint text-image semantic reasoning through a pre-trained visual language model to generate a standard response result for the query information; and pushing the standard response result to the client. This invention improves both the speed and accuracy of text and image question answering by simultaneously optimizing multiple aspects, including image encoding compression and noise reduction, unification of dual-modal feature dimensions, suppression of visual semantic interference, and multimodal joint accurate reasoning. Attached Figure Description
[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a schematic diagram illustrating the application environment of an intelligent question-answering method based on graphic-text interaction, provided as an embodiment of the present invention.
[0013] Figure 2 This is a flowchart illustrating an intelligent question-answering method based on graphic and text interaction, provided as an embodiment of the present invention.
[0014] Figure 3 This is a schematic diagram of the program modules of an intelligent question-and-answer device based on graphic and text interaction, provided in an embodiment of the present invention.
[0015] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention.
[0016] Figure 5 This is another structural schematic diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0019] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0020] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."
[0021] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0022] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0023] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0024] To illustrate the technical solution of the present invention, specific embodiments are described below.
[0025] An embodiment of the present invention provides an intelligent question-answering method based on image-text interaction, which can be applied to, for example... Figure 1 In the application environment shown, the client and server communicate via a network. The client includes, but is not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0026] Please see Figure 2 An embodiment of the present invention provides an intelligent question-answering method based on image-text interaction, wherein the method includes the following steps: S100: Obtain the query information input by the target user through the client, perform multimodal content parsing on the query information, and extract the query text and image data associated with the query text from the parsing results; S200. The query text is segmented to obtain text words, and the text words are encoded using a text Transformer encoder to generate a text word feature sequence. S300. The image data is discretized and encoded at a low level using a vector quantization encoder to generate a visual code embedding. The visual code embedding is then input into a multilayer perceptron for dimension adaptation mapping to obtain the transformed features. The transformed features are then input into a visual Transformer encoder configured with a causal attention mechanism to generate semantic visual features. S400. The semantic visual features are concatenated with the text lexical feature sequence to form a unified multimodal input sequence. Based on the unified multimodal input sequence, a pre-trained visual language large model is used to complete the joint reasoning of image and text semantics, generate a standard response result for the query information, and push the standard response result to the client.
[0027] In specific implementation, this embodiment optimizes multiple aspects, including image encoding compression and noise reduction, dual-modal feature dimension unification, visual semantic interference suppression, and multimodal joint accurate reasoning, to simultaneously improve the response speed and output accuracy of image-text question answering: Step S300 uses a vector quantization encoder to perform low-level discretization encoding on image data to generate visual code embedding, which can significantly compress the data volume of image features, reduce the amount of data for subsequent feature operations to shorten the model inference time, and improve the response output speed; a multilayer perceptron is used to complete the dimension adaptation mapping of the visual code embedding, so that the visual features and text word feature dimensions match, avoiding feature information loss caused by dimension conflicts; then, a visual Transformer encoder with a causal attention mechanism is used to model the image sequence, limiting the reverse attention interference generated by irrelevant image background blocks, and strengthening... The visual semantic representation of core targets such as medical image lesions and financial chart data is improved to reduce the negative impact of invalid image information on the reasoning results. Step S200 uses word segmentation combined with a text Transformer encoder to fully mine the professional terms and questioning intent in the query text, and obtains a semantically complete text lexical feature sequence. Step S400 can concatenate the denoised and dimension-aligned semantic visual features with the text lexical feature sequence to form a unified multimodal input sequence, which is fed into a pre-trained visual language large model to perform joint text-image semantic reasoning. This can improve the cross-modal matching degree between the text query intent and the key content of the image, reduce problems such as model illusion and irrelevant answers. While relying on vector quantization to reduce computing power and speed up the reasoning response, it also relies on pure aligned bimodal features to significantly improve the accuracy of question and answer response content.
[0028] Understandably, the intelligent question-answering method based on image-text interaction provided in this embodiment of the invention can be applied to intelligent question-answering scenarios based on image-text interaction in the medical and health field. The following is a specific example: Users upload their inquiry information through a mobile medical consultation client, entering the text inquiry "Skin erythema accompanied by itching has lasted for a week. Please check if the affected area has dermatitis characteristics and provide daily care suggestions," while simultaneously uploading a real-life image of the affected skin. Step S100 involves the system parsing the multimodal inquiry information, extracting the inquiry text and the image data of the affected area. Step S200 involves segmenting the inquiry text containing dermatology terminology into text units, which are then encoded using a text Transformer encoder to fully capture the user's request, symptom description, and other text semantics, generating a text unit feature sequence. Step S300 involves using a vector quantization encoder to perform low-level discretization encoding compression on the skin image, generating visual code embedding, and then using a multilayer perceptron to complete the process. After dimensional adaptation mapping, the input is configured with a causal attention mechanism in the visual Transformer encoder. This encoder suppresses semantic interference from irrelevant background pixels such as clothing and desktops in the image, focuses on strengthening the semantic representation of lesion areas such as erythema and papules, and outputs semantic visual features that accurately depict the characteristics of skin lesions. Step S400 is executed, where the semantic visual features of lesions are concatenated with the lexical feature sequence of the symptom description text to obtain a unified multimodal input sequence. This sequence is then input into a pre-trained medical-specific visual language model to perform image-text joint semantic reasoning. By combining the textual symptom description and the image lesion visual information, the type of skin disease is comprehensively judged, and a standardized medical Q&A response containing disease judgment, daily skin care taboos, and basic external care solutions is generated. Finally, the response is pushed to the user's mobile client for display.
[0029] Understandably, the intelligent question-answering method based on image-text interaction provided in this embodiment of the invention can also be applied to intelligent question-answering scenarios based on image-text interaction in the fintech field. The following is a specific example: Users submit multimodal inquiries through the bank's online wealth management client, inputting the text inquiry "Is the fund's return volatility too high in the past six months? Which market phases do the peak drawdowns in the chart correspond to? Is it suitable for conservative investors?" and uploading a line chart of the fund's net asset value (NAV) return. Step S100 involves the system parsing the user's submitted inquiry information, separating and extracting the wealth management inquiry text and the fund return chart image data. Step S200 involves segmenting the text containing financial terms such as drawdown, conservative investment, and return volatility to obtain text words, which are then deeply encoded using a text Transformer encoder to extract the user's core wealth management intent and generate a text word feature sequence. Step S300 involves using a vector quantization encoder to discretize and encode the fund chart image. The process involves reducing the amount of chart image feature data, then using a multilayer perceptron to perform dimensional unification mapping before feeding it into a visual Transformer encoder configured with a causal attention mechanism. This process masks invalid visual areas such as blank borders and watermarks, focusing on extracting key data visual features such as line peaks and fluctuation ranges to generate semantic visual features for financial charts. Step S400 involves concatenating the chart semantic visual features with the financial consultation text features to form a unified multimodal input sequence, which is then fed into a pre-trained visual language model for financial scenarios to perform joint text-image reasoning. This process comprehensively interprets the user's investment needs and chart return data, outputting standardized financial Q&A responses that include return fluctuation analysis, maximum drawdown interpretation, and investor type identification. The responses are then pushed to the user's financial client for display.
[0030] Furthermore, in one embodiment, the intelligent question-answering method based on image-text interaction, wherein obtaining the query information input by the target user through the client, performing multimodal content parsing on the query information, and extracting the query text and image data associated with the query text from the parsing results, specifically includes: The client responds to the target user's interface activation command, starts and displays the intelligent question and answer interface, and verifies the target user's identity information using at least one authentication method among biometrics, dynamic verification code or digital certificate on the intelligent question and answer interface. When the identity information of the target user is verified, the system receives the query information input by the target user through the intelligent question and answer interface, and performs format verification and integrity checks on the query information. When the format validation and integrity check of the query information both pass, the multimodal content of the query information is parsed, and the query text and image data associated with the query text are extracted from the parsing result. If the format verification and / or integrity check of the query information fails, a prompt message will be displayed on the intelligent question-and-answer interface.
[0031] In practical implementation, this embodiment adds multiple types of identity authentication steps before receiving user inquiry information. It verifies user identity using at least one method, such as biometrics, dynamic verification codes, or digital certificates. This effectively prevents unauthorized individuals from submitting unauthorized text and image inquiries, protecting patient medical records and image privacy in healthcare scenarios, and preventing the leakage of customer assets and document information in fintech scenarios, thus improving data interaction security. User-inputted inquiry information is only collected after identity verification is successful, and format and integrity checks are performed simultaneously. This allows for the early interception of non-compliant content such as encoding errors, corrupted images, and blank pages. Invalid query data is eliminated, avoiding meaningless calculations on invalid data in subsequent multimodal parsing, feature encoding, and large model inference processes, reducing waste of computing resources and shortening the overall response time of valid questions and answers. For queries that fail verification, prompts are pushed to the intelligent question-answering interface in real time, which can guide users to quickly correct their input and reduce the cost of interaction retry. After the verification is passed, multimodal content parsing is performed to separate and extract query text and image data, which can ensure the integrity and compliance of the data processed in the subsequent text and image dual-branch encoding stage, laying a reliable data foundation for improving the accuracy of question-answering responses and inference efficiency.
[0032] The specific implementation process of this embodiment is roughly as follows: 1. Interface loading and multi-mode identity verification After the client detects the intelligent question-and-answer interface activation command triggered by the target user, it loads and renders the intelligent question-and-answer interactive interface. After the interface is loaded, an identity verification window pops up, supporting any one of the verification channels such as biometrics, dynamic verification code, or digital certificate to collect user identity credentials. The collected credentials are uploaded to the server to complete identity comparison and verification. If the identity verification fails, the question and answer input area is locked, and the subsequent question information collection process is terminated.
[0033] 2. Compliance Inquiry Information Collection Only after the server returns an authentication successful instruction will the client unlock the text input box and image upload control, continuously receive text queries entered by the user and uploaded image files, package and integrate the text and images into a complete multimodal query data packet, and transmit it to the backend for processing.
[0034] 3. Query data packets are double-checked After receiving the multimodal query data packet, the backend performs format validation and integrity checks in parallel. The format validation checks whether the text encoding format, image suffix and file encoding are compliant. The integrity check determines whether the text is empty, whether the image file is corrupted, and whether the accompanying data is missing. If both checks pass, the multimodal parsing process begins. If any check fails, a standardized error message is generated and sent to the client.
[0035] 4. Verification of anomaly feedback and multimodal data extraction branch If any verification fails, the client receives a prompt and displays the corresponding correction prompt in a pop-up window on the intelligent Q&A interface, guiding the user to re-edit or upload the data; if the format and integrity verifications all pass, the backend performs multimodal content parsing on the data packet, separates and peels off the plain text information stream and the image binary data stream, extracts the query text and the accompanying image data respectively, and outputs them to the subsequent text encoding and image encoding branches.
[0036] Furthermore, in one embodiment, the intelligent question-answering method based on image-text interaction, wherein the step of segmenting the query text to obtain text units, and encoding the text units using a text Transformer encoder to generate a text unit feature sequence, specifically includes: The query text is cleaned and noise removed to obtain intermediate text data; Based on a multi-granularity hybrid word segmentation strategy, the intermediate text data is segmented into words to generate an initial word sequence. Then, a preset text modality embedding identifier is concatenated to the beginning of the initial word sequence to obtain text words. The text lexical units are input into the text Transformer encoder. The dynamic sparse self-attention mask built into the text Transformer encoder adaptively masks the low-association lexical units at a distance, and the semantic associations of the context are mined layer by layer and the encoding is completed to generate a text lexical feature sequence.
[0037] Furthermore, the intelligent question-answering method based on image-text interaction, wherein the step of using a vector quantization encoder to perform low-level discretization encoding on the image data to generate visual code embeddings, inputting the visual code embeddings into a multilayer perceptron for dimension adaptation mapping to obtain transformed features, and inputting the transformed features into a visual Transformer encoder configured with a causal attention mechanism to generate semantic visual features, specifically includes: The image data is subjected to foreground effective region weight filtering processing to remove low-weight invalid background pixels, thereby obtaining the target image data; The target image data is discretized and encoded at a low level using a vector quantization encoder, and a weighted visual code embedding is generated by combining the codebook weighting strategy for the image-text question-and-answer scenario. The weighted visual code is embedded into a multilayer perceptron, and a cross-modal dimension normalization mapping strategy is used to complete the dimension adaptation mapping of the weighted visual code embedding, generating transformed features. The transformed features are input into a visual Transformer encoder configured with a causal attention mechanism. The reverse visual block dependency is blocked by the image sequence unidirectional constraint mask built into the visual Transformer encoder, and the local and global visual semantic associations are extracted layer by layer to generate semantic visual features.
[0038] In specific implementation, this embodiment first cleanses and removes invalid noise from the query text processing flow. Then, it uses multi-granularity hybrid word segmentation to generate an initial word sequence and pre-assembles text modality embedding identifiers. Relying on the dynamic sparse self-attention mask built into the text Transformer encoder, it adaptively masks long-distance, low-association redundant words. This can accurately capture the subdivided granular terminology and core query intent within medical and financial professional texts, reduce computational redundancy caused by invalid words, and improve text encoding speed. The modality embedding identifiers can also provide a clear basis for distinguishing text boundaries for subsequent multimodal fusion. For image data, it first performs foreground effective region weight screening to remove irrelevant background pixels. Then, it uses a vector quantization encoder to achieve discrete compression of the image's low-level features and combines scene codebook weighting strategies to enhance the features of lesions, financial charts, etc. The key visual target representation leverages the cross-modal dimensional normalization mapping of a multilayer perceptron to unify the image and text feature dimensions, thereby eliminating modal numerical offsets. Finally, a causal attention visual Transformer encoder equipped with a unidirectional constraint mask for image sequences blocks invalid inverse visual block dependencies and suppresses background noise interference, significantly compressing the amount of image feature data to reduce inference computational overhead and accelerate response speed. Simultaneously, effective semantic features are extracted from both the text and image ends to weaken irrelevant information interference, allowing subsequent multimodal splicing and joint inference to be carried out based on high-purity, dimension-matched, and clearly defined bimodal features. This significantly improves the cross-modal semantic alignment accuracy between text query intent and key image information, effectively alleviating model illusion and irrelevant answering problems, and simultaneously achieving a dual improvement in computational power optimization and question-and-answer response accuracy during the image and text encoding stage.
[0039] The specific implementation process of this embodiment is roughly as follows: 1) Query text encoding process 1. Text noise reduction preprocessing Read the original query text submitted by the user, perform data cleaning and noise removal operations, filter out meaningless symbols, repeated interjections, invalid placeholder characters, garbled text and other interfering information, remove redundant content in the text that has no semantic value, and output the cleaned intermediate text data.
[0040] 2. Multi-granularity word segmentation and modal tagging concatenation A multi-granularity hybrid word segmentation strategy that simultaneously covers character granularity and word granularity is adopted to segment the intermediate text data into lexical units and output an ordered initial lexical unit sequence. A predefined text modality embedding identifier vector is retrieved and concatenated at the beginning of the initial lexical unit sequence to form a complete text lexical unit sequence.
[0041] 3. Text Transformer Encoding with Dynamic Sparse Masking The concatenated text lexical units are input into the text Transformer encoder. The encoder generates a dynamic sparse self-attention mask based on the text length and keyword weights. It automatically filters out redundant lexical units that are far from the core question and have low relevance, and only retains highly relevant lexical units for attention calculation. It then mines the semantic relationships of the context layer by layer and completes feature encoding, finally outputting a text lexical feature sequence with complete semantics and less redundant information.
[0042] 2) Image semantic encoding process 1. Image foreground selection preprocessing Read the raw image data uploaded by the user, perform foreground effective region weight filtering, calculate the semantic weight of each pixel region in the image, remove low-weight blank background and irrelevant clutter pixels, retain only pixel regions containing key targets such as medical lesions and financial charts, and output the simplified target image data.
[0043] 2. Vector Quantization Coding and Scene-Weighted Embedding The target image data is fed into a vector quantization encoder to extract low-level visual features and complete low-level discretization encoding. Combined with the weighting strategy of the codebook for medical and financial question-and-answer scenarios, higher weights are given to the quantization index vector of the corresponding professional target to generate a weighted visual code embedding that highlights the key visual target.
[0044] 3. Cross-modal normalized dimensionality adaptation mapping By embedding weighted visual codes into the input multilayer perceptron and enabling a cross-modal dimension normalization mapping strategy, the feature dimensions of visual features and text lexical features are unified. At the same time, the numerical distribution differences between the two modalities are corrected, and the transformed features with dimension matching and balanced distribution are output.
[0045] 4. Causal Attention Visual Transformer Deep Encoding The transformed features are input into a visual Transformer encoder equipped with a causal attention mechanism. The encoder relies on a built-in image sequence unidirectional constraint mask to prevent subsequent image blocks from establishing attention dependencies with the preceding blocks, blocking ineffective visual associations in the reverse direction, extracting local details and global semantics layer by layer, and outputting semantic visual features that are resistant to background interference and enhance the representation of key targets.
[0046] Furthermore, in one embodiment, the intelligent question-answering method based on image-text interaction, wherein concatenating the semantic visual features with the text lexical feature sequence to form a unified multimodal input sequence, and performing joint image-text semantic reasoning based on the unified multimodal input sequence through a pre-trained visual language large model to generate a standard response result for the query information, and pushing the standard response result to the client, specifically includes: The feature distributions of the semantic visual features and the text lexical feature sequences are corrected respectively to eliminate modality shift, and the corrected semantic visual features and text lexical feature sequences are concatenated according to a preset concatenation strategy to form a unified multimodal input sequence; The unified multimodal input sequence is input into a pre-trained visual language model. The visual language model performs joint computation of image and text semantics based on question-answering guidance and hierarchical reasoning to generate a preliminary response result to the query information. The preliminary response result is standardized to obtain a standard response result for the query information, and the standard response result is pushed to the client.
[0047] Furthermore, the intelligent question-answering method based on image-text interaction, wherein the step of correcting the feature distribution of the semantic visual features and the text lexical feature sequence to eliminate modality shift, and concatenating the corrected semantic visual features and the text lexical feature sequence according to a preset concatenation strategy to form a unified multimodal input sequence, specifically includes: Modal adaptive normalization correction is performed on the semantic visual features and the text word feature sequence. The mean and variance of the semantic visual features and the text word feature sequence are calculated and distributed translation and scaling are performed. Corresponding modal distribution deviation labels are generated synchronously to eliminate cross-modal numerical offset between the semantic visual features and the text word feature sequence. Obtain the visual modality embedding identifier carried by the corrected semantic visual features and the text modality embedding identifier carried by the corrected text lexical feature sequence. Embed the visual modality embedding identifier and the text modality embedding identifier into the beginning of the corresponding feature sequence respectively. Then, complete the splicing of the corrected semantic visual features and the text lexical feature sequence according to the preset splicing strategy to generate a unified multimodal input sequence.
[0048] Furthermore, the intelligent question-answering method based on image-text interaction, wherein standardizing the preliminary response result to obtain a standard response result for the query information, and pushing the standard response result to the client, specifically includes: The preliminary response results are then subjected to answer verification, evidence tracing and annotation, and content organization to generate intermediate response results; Based on the client's configuration information, the intermediate response result is formatted to obtain a standard response result for the query information; The standard response result is pushed to the client and displayed to the target user on the client according to a preset result display strategy.
[0049] In specific implementation, this embodiment first performs modal adaptive normalization correction on semantic visual features and text lexical feature sequences respectively. It eliminates numerical offsets between image and text modalities by mean-variance shifting and scaling, while retaining modal distribution deviation markers. Then, relying on their respective modal embedding identifiers, it distinguishes between the two modalities and sequentially concatenates them according to fixed rules to form a unified multimodal input sequence. This eliminates semantic matching bias caused by differences in feature distribution across different modalities. The modal identifiers enable the visual language large model to accurately distinguish between text query semantics and image visual information, significantly improving cross-modal semantic alignment. The normalized multimodal sequence is then fed into the pre-trained visual language large model to perform question-answering-oriented hierarchical image-text joint reasoning. It hierarchically matches key image and text information around the user's core query intent, reducing model illusions and... The system reduces the probability of incorrect answers and outputs more reliable preliminary responses. Subsequent steps involve multi-layered answer verification to filter out errors, complete evidence tracing and annotation (if any evidence documents are cited), and refine the answer content to obtain intermediate responses. The display format is then adaptively adjusted based on different client device configurations. Finally, the responses are pushed out and displayed in layers according to a customized strategy. This approach meets the regulatory requirements for traceable results and rigorous content compliance in the medical and financial fields, while also adapting to the differentiated rendering needs of multiple terminals such as mobile phones and PCs. Furthermore, the entire multi-modal correction, structured reasoning, and standardized output process refines effective features and filters out invalid information, reducing redundant computation in large models and simultaneously improving the reasoning response speed, accuracy, readability, and scenario adaptability of the text-based question-and-answer system.
[0050] The specific implementation process of this embodiment is roughly as follows: 1. Adaptive normalization correction processing for dual-modal feature sub-modes Semantic visual features and text lexical feature sequences output from upstream are read separately, and adaptive normalization correction is performed independently for the two modalities. The mean and variance of all vectors within a single feature are calculated separately, and the numerical distribution range of the two types of features is unified through distribution translation and scaling transformation to eliminate the numerical offset problem between the visual and text modalities. During the correction operation, distribution deviation markers specific to each modality are recorded and generated simultaneously for subsequent inference and tracing.
[0051] 2. Modal identifier embedding and ordered splicing of multimodal sequences After the semantic visual features are extracted and corrected, the visual modality embedding identifiers corresponding to the semantic visual features and the text modality embedding identifiers corresponding to the text word feature sequences are extracted and added to the beginning of the corresponding feature sequences. The two sets of features are combined according to the preset splicing strategy of text sequence before visual features, and a complete unified multimodal input sequence with clear boundaries and uniform dimensions is generated.
[0052] 3. Question-and-answer oriented hierarchical text and image joint reasoning generates preliminary responses. The concatenated unified multimodal input sequence is input into the pre-trained visual language model; the model executes question-answering-oriented hierarchical reasoning logic. The first layer completes cross-modal matching and alignment of the semantics of the user's query text with the key visual features of the image. The second layer mines the image-text association information around the user's core question intent and infers the response content. Finally, it outputs the preliminary response result that has not been regularized and verified.
[0053] 4. Preliminary response verification and source tracing, and terminal adaptive format standardization. The initial response results output by the model are subjected to multi-layer answer verification, which verifies the consistency of image and text matching, filters out violations and incorrect expressions, completes evidence source annotation (if there are cited evidence documents), removes redundant sentences to complete content regularization, and generates intermediate response results; reads the device configuration parameters of the currently connected client, and adaptively adjusts the image and text layout and text format of the intermediate response based on screen size, image and text rendering rules, and layout parameters to generate standard response results adapted to the client.
[0054] 5. Standard reply push and client-side tiered display The server pushes the packaged standard response data packet to the client that initiated the inquiry; the client renders the content in layers according to the preset result display strategy, dividing the content into core answer blocks and image and text traceability annotation blocks, and fully displaying the standardized Q&A response content to the target user.
[0055] As can be seen from the above method embodiments, the intelligent question-answering method based on image-text interaction provided by the present invention includes: acquiring query information input by a target user through a client; performing multimodal content parsing on the query information; extracting query text and image data associated with the query text from the parsing results; performing word segmentation on the query text to obtain text units, and encoding the text units using a text Transformer encoder to generate a text unit feature sequence; performing low-level discretization encoding on the image data using a vector quantization encoder to generate visual code embeddings; inputting the visual code embeddings into a multilayer perceptron for dimension adaptation mapping to obtain transformed features; inputting the transformed features into a visual Transformer encoder configured with a causal attention mechanism to generate semantic visual features; concatenating the semantic visual features with the text unit feature sequence to form a unified multimodal input sequence; and based on the unified multimodal input sequence, performing joint image-text semantic reasoning through a pre-trained visual language large model to generate a standard response result for the query information; and pushing the standard response result to the client. The method of this invention can simultaneously improve the speed and accuracy of text and image question answering by optimizing multiple aspects such as image encoding compression and noise reduction, dual-modal feature dimension unification, visual semantic interference suppression, and multimodal joint accurate reasoning.
[0056] It should be understood that although this application provides the method operation steps as described in the embodiments or flowcharts, conventional or non-inventive labor may include more or fewer operation steps, and these operation steps are not necessarily executed sequentially according to the order of the embodiments or flowcharts. The order of steps listed in the embodiments or flowcharts is merely one way of executing many steps and does not represent the only execution order. It should be noted that there is no necessary sequential order between the above steps. Those skilled in the art can understand from the description of the embodiments of the present invention that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in exchange, etc. Moreover, at least some steps in the embodiments or flowcharts may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn, alternately, or synchronously with other steps or at least a part of the sub-steps or stages of other steps.
[0057] Based on the above method embodiments, please refer to Figure 3 Another embodiment of the present invention also provides an intelligent question-answering device based on graphic and text interaction, wherein the device includes: The acquisition module 11 is used to acquire the query information input by the target user through the client, perform multimodal content parsing on the query information, and extract the query text and image data associated with the query text from the parsing results; The first encoding module 12 is used to perform word segmentation on the query text to obtain text words, and to encode the text words using a text Transformer encoder to generate a text word feature sequence. The second encoding module 13 is used to perform low-level discretization encoding on the image data using a vector quantization encoder to generate a visual code embedding, input the visual code embedding into a multilayer perceptron for dimension adaptation mapping to obtain transformed features, and input the transformed features into a visual Transformer encoder configured with a causal attention mechanism to generate semantic visual features. The splicing module 14 is used to splice the semantic visual features and the text word feature sequence to form a unified multimodal input sequence, and based on the unified multimodal input sequence, to perform joint reasoning of image and text semantics through a pre-trained visual language big model, generate a standard response result for the query information, and push the standard response result to the client.
[0058] Furthermore, in one embodiment, the intelligent question-answering device based on image-text interaction, wherein obtaining the query information input by the target user through a client, performing multimodal content parsing on the query information, and extracting the query text and image data associated with the query text from the parsing results specifically includes: The client responds to the target user's interface activation command, starts and displays the intelligent question and answer interface, and verifies the target user's identity information using at least one authentication method among biometrics, dynamic verification code or digital certificate on the intelligent question and answer interface. When the identity information of the target user is verified, the system receives the query information input by the target user through the intelligent question and answer interface, and performs format verification and integrity checks on the query information. When the format validation and integrity check of the query information both pass, the multimodal content of the query information is parsed, and the query text and image data associated with the query text are extracted from the parsing result. If the format verification and / or integrity check of the query information fails, a prompt message will be displayed on the intelligent question-and-answer interface.
[0059] Furthermore, in one embodiment, the intelligent question-answering device based on image-text interaction, wherein the step of segmenting the query text to obtain text units, and encoding the text units using a text Transformer encoder to generate a text unit feature sequence, specifically includes: The query text is cleaned and noise removed to obtain intermediate text data; Based on a multi-granularity hybrid word segmentation strategy, the intermediate text data is segmented into words to generate an initial word sequence. Then, a preset text modality embedding identifier is concatenated to the beginning of the initial word sequence to obtain text words. The text lexical units are input into the text Transformer encoder. The dynamic sparse self-attention mask built into the text Transformer encoder adaptively masks the low-association lexical units at a distance, and the semantic associations of the context are mined layer by layer and the encoding is completed to generate a text lexical feature sequence.
[0060] Furthermore, in the aforementioned intelligent question-answering device based on image-text interaction, the step of using a vector quantization encoder to perform low-level discretization encoding on the image data to generate visual code embeddings, inputting the visual code embeddings into a multilayer perceptron for dimension adaptation mapping to obtain transformed features, and inputting the transformed features into a visual Transformer encoder configured with a causal attention mechanism to generate semantic visual features, specifically includes: The image data is subjected to foreground effective region weight filtering processing to remove low-weight invalid background pixels, thereby obtaining the target image data; The target image data is discretized and encoded at a low level using a vector quantization encoder, and a weighted visual code embedding is generated by combining the codebook weighting strategy for the image-text question-and-answer scenario. The weighted visual code is embedded into a multilayer perceptron, and a cross-modal dimension normalization mapping strategy is used to complete the dimension adaptation mapping of the weighted visual code embedding, generating transformed features. The transformed features are input into a visual Transformer encoder configured with a causal attention mechanism. The reverse visual block dependency is blocked by the image sequence unidirectional constraint mask built into the visual Transformer encoder, and the local and global visual semantic associations are extracted layer by layer to generate semantic visual features.
[0061] Furthermore, in one embodiment, the intelligent question-answering device based on image-text interaction, wherein concatenating the semantic visual features with the text lexical feature sequence to form a unified multimodal input sequence, and based on the unified multimodal input sequence, performing joint image-text semantic reasoning through a pre-trained visual language large model to generate a standard response result for the query information, and pushing the standard response result to the client, specifically includes: The feature distributions of the semantic visual features and the text lexical feature sequences are corrected respectively to eliminate modality shift, and the corrected semantic visual features and text lexical feature sequences are concatenated according to a preset concatenation strategy to form a unified multimodal input sequence; The unified multimodal input sequence is input into a pre-trained visual language model. The visual language model performs joint computation of image and text semantics based on question-answering guidance and hierarchical reasoning to generate a preliminary response result to the query information. The preliminary response result is standardized to obtain a standard response result for the query information, and the standard response result is pushed to the client.
[0062] Furthermore, in the aforementioned intelligent question-answering device based on image-text interaction, the step of respectively correcting the feature distribution of the semantic visual features and the text lexical feature sequence to eliminate modality shift, and then concatenating the corrected semantic visual features and the text lexical feature sequence according to a preset concatenation strategy to form a unified multimodal input sequence, specifically includes: Modal adaptive normalization correction is performed on the semantic visual features and the text word feature sequence. The mean and variance of the semantic visual features and the text word feature sequence are calculated and distributed translation and scaling are performed. Corresponding modal distribution deviation labels are generated synchronously to eliminate cross-modal numerical offset between the semantic visual features and the text word feature sequence. Obtain the visual modality embedding identifier carried by the corrected semantic visual features and the text modality embedding identifier carried by the corrected text lexical feature sequence. Embed the visual modality embedding identifier and the text modality embedding identifier into the beginning of the corresponding feature sequence respectively. Then, complete the splicing of the corrected semantic visual features and the text lexical feature sequence according to the preset splicing strategy to generate a unified multimodal input sequence.
[0063] Furthermore, in the aforementioned intelligent question-and-answer device based on text and image interaction, the standardization process of the preliminary response result to obtain a standard response result for the query information, and the push of the standard response result to the client, specifically includes: The preliminary response results are then subjected to answer verification, evidence tracing and annotation, and content organization to generate intermediate response results; Based on the client's configuration information, the intermediate response result is formatted to obtain a standard response result for the query information; The standard response result is pushed to the client and displayed to the target user on the client according to a preset result display strategy.
[0064] It should be noted that, in the device embodiments of the present invention, the information interaction and execution process between the above modules are based on the same concept as in the method embodiments of the present invention. For details on their specific functions and the resulting technical effects, please refer to the aforementioned method embodiments section, which will not be repeated here.
[0065] Based on the above method embodiments, another embodiment of the present invention also provides a computer device, which can be a server, and its internal structure diagram can be as follows. Figure 4 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the intelligent question-answering method based on text-image interaction as described in any of the above method embodiments.
[0066] Based on the above method embodiments, another embodiment of the present invention also provides a computer device, which can be a client, and its internal structure diagram can be as follows. Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements the client-side functions or steps of the intelligent question-answering method based on text-image interaction as described in any of the above method embodiments.
[0067] Those skilled in the art will understand that Figure 4 and Figure 5The structural schematic diagram shown is only a schematic diagram of a part of the structure related to the present invention and does not constitute a limitation on the computer device on which the present invention is applied. The specific computer device may include more components than shown in the figure, or combine certain components, or have different component arrangements.
[0068] The processor referred to herein can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.
[0069] The memory includes readable storage media, internal memory, etc., where internal memory can be the RAM of a computer device. Internal memory provides an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of the computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal storage units and external storage devices of the computer device. The memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. The memory can also be used to temporarily store data that has been output or will be output.
[0070] Based on the above method embodiments, another embodiment of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the intelligent question-answering method based on image-text interaction as described in any of the above method embodiments. The computer-readable storage medium may be non-volatile or volatile.
[0071] It should be noted that the functions or steps that can be achieved by the computer-readable storage medium or computer device, and the technical effects brought about by the functions / steps, can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0072] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc. The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.
[0073] Those skilled in the art will understand that, for the sake of convenience and brevity, the embodiments of the device of the present invention are only illustrated by the division of the above-mentioned functional units and modules. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the present invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
[0074] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0075] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0076] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0077] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.
[0078] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. An intelligent question-answering method based on image-text interaction, characterized in that, include: The system obtains query information input by the target user through the client, performs multimodal content parsing on the query information, and extracts the query text and image data associated with the query text from the parsing results. The query text is segmented to obtain text words, and the text words are encoded using a text Transformer encoder to generate a text word feature sequence. The image data is discretized and encoded at a low level using a vector quantization encoder to generate a visual code embedding. The visual code embedding is then input into a multilayer perceptron for dimension adaptation mapping to obtain transformed features. The transformed features are then input into a visual Transformer encoder configured with a causal attention mechanism to generate semantic visual features. The semantic visual features are concatenated with the text lexical feature sequence to form a unified multimodal input sequence. Based on the unified multimodal input sequence, a pre-trained visual language large model is used to perform joint text-image semantic reasoning to generate a standard response result for the query information. The standard response result is then pushed to the client.
2. The intelligent question-answering method based on image-text interaction according to claim 1, characterized in that, The process of obtaining query information input by the target user through a client, performing multimodal content parsing on the query information, and extracting query text and image data associated with the query text from the parsing results includes: The client responds to the target user's interface activation command, starts and displays the intelligent question and answer interface, and verifies the target user's identity information using at least one authentication method among biometrics, dynamic verification code or digital certificate on the intelligent question and answer interface. When the identity information of the target user is verified, the system receives the query information input by the target user through the intelligent question and answer interface, and performs format verification and integrity checks on the query information. When the format validation and integrity check of the query information both pass, the multimodal content of the query information is parsed, and the query text and image data associated with the query text are extracted from the parsing result. If the format verification and / or integrity check of the query information fails, a prompt message will be displayed on the intelligent question-and-answer interface.
3. The intelligent question-answering method based on image-text interaction according to claim 1, characterized in that, The step of segmenting the query text to obtain text tokens, and then encoding the text tokens using a text Transformer encoder to generate a text token feature sequence includes: The query text is cleaned and noise removed to obtain intermediate text data; Based on a multi-granularity hybrid word segmentation strategy, the intermediate text data is segmented into words to generate an initial word sequence. Then, a preset text modality embedding identifier is concatenated to the beginning of the initial word sequence to obtain text words. The text lexical units are input into the text Transformer encoder. The dynamic sparse self-attention mask built into the text Transformer encoder adaptively masks the low-association lexical units at a distance, and the semantic associations of the context are mined layer by layer and the encoding is completed to generate a text lexical feature sequence.
4. The intelligent question-answering method based on image-text interaction according to claim 1, characterized in that, The image data is discretized using a vector quantization encoder to generate a visual code embedding. This visual code embedding is then input into a multilayer perceptron for dimensionality adaptation mapping to obtain transformed features. These transformed features are then input into a visual Transformer encoder configured with a causal attention mechanism to generate semantic visual features, including: The image data is subjected to foreground effective region weight filtering processing to remove low-weight invalid background pixels, thereby obtaining the target image data; The target image data is discretized and encoded at a low level using a vector quantization encoder, and a weighted visual code embedding is generated by combining the codebook weighting strategy for the image-text question-and-answer scenario. The weighted visual code is embedded into a multilayer perceptron, and a cross-modal dimension normalization mapping strategy is used to complete the dimension adaptation mapping of the weighted visual code embedding, generating transformed features. The transformed features are input into a visual Transformer encoder configured with a causal attention mechanism. The reverse visual block dependency is blocked by the image sequence unidirectional constraint mask built into the visual Transformer encoder, and the local and global visual semantic associations are extracted layer by layer to generate semantic visual features.
5. The intelligent question-answering method based on image-text interaction according to claim 1, characterized in that, The process of concatenating the semantic visual features with the text lexical feature sequence to form a unified multimodal input sequence, and based on the unified multimodal input sequence, performing joint text-image semantic reasoning through a pre-trained visual language model to generate a standard response result for the query information, and pushing the standard response result to the client, includes: The feature distributions of the semantic visual features and the text lexical feature sequences are corrected respectively to eliminate modality shift, and the corrected semantic visual features and text lexical feature sequences are concatenated according to a preset concatenation strategy to form a unified multimodal input sequence; The unified multimodal input sequence is input into a pre-trained visual language model. The visual language model performs joint computation of image and text semantics based on question-answering guidance and hierarchical reasoning to generate a preliminary response result to the query information. The preliminary response result is standardized to obtain a standard response result for the query information, and the standard response result is pushed to the client.
6. The intelligent question-answering method based on image-text interaction according to claim 5, characterized in that, The step of correcting the feature distributions of the semantic visual features and the text lexical feature sequences to eliminate modality shift, and then concatenating the corrected semantic visual features and text lexical feature sequences according to a preset concatenation strategy to form a unified multimodal input sequence includes: Modal adaptive normalization correction is performed on the semantic visual features and the text word feature sequence. The mean and variance of the semantic visual features and the text word feature sequence are calculated and distributed translation and scaling are performed. Corresponding modal distribution deviation labels are generated synchronously to eliminate cross-modal numerical offset between the semantic visual features and the text word feature sequence. Obtain the visual modality embedding identifier carried by the corrected semantic visual features and the text modality embedding identifier carried by the corrected text lexical feature sequence. Embed the visual modality embedding identifier and the text modality embedding identifier into the beginning of the corresponding feature sequence respectively. Then, complete the splicing of the corrected semantic visual features and the text lexical feature sequence according to the preset splicing strategy to generate a unified multimodal input sequence.
7. The intelligent question-answering method based on image-text interaction according to claim 5, characterized in that, The standardization process for the preliminary response result to obtain a standard response result for the inquiry information, and the push of the standard response result to the client, includes: The preliminary response results are then subjected to answer verification, evidence tracing and annotation, and content organization to generate intermediate response results; Based on the client's configuration information, the intermediate response result is formatted to obtain a standard response result for the query information; The standard response result is pushed to the client and displayed to the target user on the client according to a preset result display strategy.
8. An intelligent question-and-answer device based on graphic and text interaction, characterized in that, include: The acquisition module is used to acquire the query information input by the target user through the client, perform multimodal content parsing on the query information, and extract the query text and image data associated with the query text from the parsing results; The first encoding module is used to perform word segmentation on the query text to obtain text words, and to encode the text words using a text Transformer encoder to generate a text word feature sequence. The second encoding module is used to perform low-level discretization encoding on the image data using a vector quantization encoder to generate a visual code embedding. The visual code embedding is then input into a multilayer perceptron for dimension adaptation mapping to obtain transformed features. The transformed features are then input into a visual Transformer encoder configured with a causal attention mechanism to generate semantic visual features. The concatenation module is used to concatenate the semantic visual features with the text lexical feature sequence to form a unified multimodal input sequence, and based on the unified multimodal input sequence, to perform joint text-image semantic reasoning through a pre-trained visual language large model, generate a standard response result for the query information, and push the standard response result to the client.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the intelligent question-answering method based on graphic-text interaction as described in any one of claims 1-7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the intelligent question-answering method based on graphic-text interaction as described in any one of claims 1-7.
