Question answering method and apparatus, and device and storage medium

By capturing image data and obtaining extended information, and determining answers based on the knowledge base, the accuracy of multimodal visual language Q&A application in the event of incomplete image data or poor quality is solved, and convenient Q&A services for visually impaired people are realized, improving The efficiency and accuracy of Q&A services.

WO2024227415A1PCT designated stage expired Publication Date: 2024-11-07BEIJING YOUZHUJU NETWORK TECH CO LTD

Patent Information

Application Number
PCT/CN2024/089651
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-04
Filing Date
2024-04-24
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

Existing multimodal visual language Q&A applications are difficult to generate accurate answers when processing image data of poor or incomplete quality. Especially for visually impaired people, it is difficult to detect whether the quality of image data meets the Q&A needs, resulting in Q&A service efficiency and Reduced accuracy.

Method used

By capturing image data, extracting text information, and obtaining extended information associated with text information, determining the target answers to the question based on image data and extended information, introducing a knowledge base to expand the accurate answering ability, which is suitable for multimodal data scenarios. In particular, instant and accurate Q&A services are provided when the image data is incomplete or insufficient.

Benefits of technology

In the case of poor or incomplete image data quality, it can effectively assist users in realizing multimodal visual language Q&A, improve the accuracy and efficiency of Q&A services, especially to provide convenience for people with vision impairment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024089651_07112024_PF_FP_ABST
    Figure CN2024089651_07112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided are a question answering method and apparatus, and a device and a storage medium. The question answering method comprises: in response to a question answering initiation operation being detected, using a device of a user to capture image data and a question for the image data (210); extracting text information from the image data (220); acquiring expanded information associated with the text information (230); and determining a target answer for the question on the basis of the image data and the expanded information (240). In this way, in a question answering scenario of multi-modal data, a knowledge base can be introduced to expand the capability of accurately answering a question. Thus, an instant and accurate question answering service can still be provided for a user even if image data is incomplete and insufficient.
Need to check novelty before this filing date? Find Prior Art

Description

Method, device, equipment and storage medium for question answering

[0001] This application claims priority to the Chinese invention patent application entitled “Method, apparatus, device and storage medium for question and answer” filed on May 4, 2023, with application number: 202310492380.4. The entire contents of that application are incorporated by reference into this application. Technical Field

[0002] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for question answering. Background Art

[0003] With the rapid development of information technology, more and more applications are offering question-and-answer (Q&A) functionality, bringing significant benefits to users. Applications with Q&A functionality can output responses based on user input, whether in voice or text. Applications with multimodal Visual Question Answering (VQA) functionality can also generate audio responses tailored to user input images, based on voice questions.

[0004] Summary of the Invention

[0005] In a first aspect of the present disclosure, a question-and-answer method is provided. The method includes: in response to detecting a question-and-answer initiation operation, capturing image data and a question related to the image data using a user's device; extracting text information from the image data; obtaining extended information associated with the text information; and determining a target answer to the question based on the image data and the extended information.

[0006] In a second aspect of the present disclosure, a device for question-and-answering is provided. The device includes: a data capture module configured to, in response to detecting a question-and-answer initiation operation, capture image data and a question related to the image data using a user's device; a text information extraction module configured to extract text information from the image data; an extended information acquisition module configured to acquire extended information associated with the text information; and a target answer determination module configured to determine a target answer to the question based on the image data and the extended information.

[0007] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the electronic device to perform the method of the first aspect.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer-executable instructions, which, when executed by a device, cause the device to perform the method of the first aspect.

[0010] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] FIG1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0013] FIG2 shows a flowchart of a question-and-answer process according to some embodiments of the present disclosure;

[0014] FIG3 shows a schematic diagram of a question-and-answer interface according to some embodiments of the present disclosure;

[0015] FIG4 is a schematic diagram showing a question-answering process according to some embodiments of the present disclosure;

[0016] FIG5 shows a schematic structural block diagram of an apparatus for applying question-answering according to some embodiments of the present disclosure; and

[0017] FIG6 illustrates a block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0018] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0019] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and / or to be developed in the future.

[0020] Herein, unless explicitly stated otherwise, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0021] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0022] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0023] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0024] As an optional but non-limiting embodiment, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also include a target exploration control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0025] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the embodiments of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the embodiments of the present disclosure.

[0026] As used herein, the term "model" can learn the association between corresponding inputs and outputs from training data, so that after training is completed, corresponding outputs can be generated for given inputs. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0027] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs. It typically includes an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the previous layer.

[0028] Generally speaking, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also known as the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also known as input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. In the application stage, the model can be used to process the actual input based on the parameter values ​​obtained through training to determine the corresponding output.

[0029] As briefly mentioned above, an application with a multimodal visual language question-answering function can play an audio answer to an image input by a user based on a voice question input by the user.

[0030] Visual-language question answering (VQA) is a multimodal understanding task that requires answering verbal questions based on understanding visual content. Traditionally, applications with multimodal VQA capabilities have leveraged trained question-answering models to implement VQA.

[0031] However, in some cases, the image data collected by the user may have problems such as poor quality and incomplete content. For example, the image data captured by the user is relatively blurry, the part of the image data that the user wants to ask a question is not fully captured, or some text in the image data is in a small font. Based on such image data, it may be difficult to generate a correct answer from it. In particular, for certain groups of people, such as the visually impaired, there is a greater demand for visual language question answering. However, due to their limited vision, these people often find it more difficult to detect whether the quality of the captured image data meets the question-answering needs. Therefore, it is expected that in question-answering scenarios based on visual data, accurate answers can still be obtained even based on poor-quality image data.

[0032] Embodiments of the present disclosure provide an improved Q&A solution. According to this solution, image data is collected and questions about the image data are indicated. Extended information associated with the textual information in the image data is obtained. Answers to the questions are determined based on the image data and the extended information. In this way, a knowledge base can be introduced in multimodal data Q&A scenarios to expand the ability to accurately answer questions. This allows users to be provided with instant and accurate Q&A services even when image data is incomplete or insufficient.

[0033] Furthermore, the question-and-answer solution proposed in this disclosure can effectively assist users, especially those with persistent or temporary vision impairment or impairment, by accurately implementing multimodal visual language question-and-answer. It should be understood that the solution provided by the embodiments of this disclosure can provide convenience for specific groups of people, but this does not imply any discrimination against specific groups of people.

[0034] FIG1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. In this example environment 100, an application 120 is installed on a terminal device 110. A user 140 can interact with the application 120 via the terminal device 110 and / or its attached devices. The application 120 is an application having at least a question-and-answer function.

[0035] In some embodiments, the terminal device 110 communicates with the server 130 to provide services for the application 120. The terminal device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the terminal device 110 can also support any type of interface for the user (such as "wearable" circuitry, etc.).

[0036] The terminal device 110 may, for example, include a sensor of an appropriate type for detecting user gestures. For example, the terminal device 110 may include a touch screen for detecting various types of gestures made by the user on the touch screen. Alternatively or additionally, the terminal device 110 may also include other appropriate types of sensing devices such as proximity sensors to detect various types of gestures made by the user within a predetermined distance above the screen. The terminal device 110 may also include, for example, a sound collection device (such as a microphone) for collecting user audio, a sound playback device (such as a speaker) for playing audio, an image collection device (such as a camera, a webcam, etc.) for collecting images, and a display device (such as a display screen, which may be a touch screen) for displaying an interface, etc.

[0037] Server 130 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. Server 130 can include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in a cloud environment, and the like. Server 130 can provide backend services for application 120 in terminal device 110.

[0038] In some embodiments discussed below, the question-and-answer function can be implemented using multiple models with various functions. One or more of these models can be remotely deployed in the server 130, and the terminal device 110 can utilize these multiple models to implement the corresponding functions through communication with the server 130. This can save resources and power of the terminal device 110 and utilize the powerful resources of the server to improve computing efficiency. In some embodiments, one or more of these models can also be deployed locally on the terminal device 110. This can be selected based on actual circumstances.

[0039] In some embodiments, in the environment 100 of FIG. 1 , if the application 120 is active, the terminal device 110 may present an interface 150 of the application 120. Through the interface 150, the application 120 may provide the user 140 with one or more services related to the question-and-answer function, including voice capture, image capture, voice playback, text display, and the like.

[0040] It should be understood that the structure and functionality of environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0041] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0042] 2 shows a flow chart of a question-answering process 200 according to some embodiments of the present disclosure. The process 200 may be implemented at the terminal device 110. For ease of discussion, the process 200 will be described with reference to the environment 100 of FIG. 1 .

[0043] In block 210 , the terminal device 110 captures image data and a question regarding the image data in response to detecting a question-and-answer initiation operation.

[0044] In some embodiments, the terminal device 110 may capture image data and indicate questions about the image data in response to detecting a question-and-answer initiation operation. Specifically, the terminal device 110 directly detects the question-and-answer initiation operation initiated by the user. For example, the terminal device 110 may determine that a question-and-answer initiation operation is detected in response to detecting a question-and-answer initiation voice (e.g., "turn on the question-and-answer function"). For another example, the terminal device 110 may determine that a question-and-answer initiation operation is detected in response to detecting a preset operation on a hardware button (e.g., a press operation, a long press operation, etc.). In some embodiments, after detecting the question-and-answer initiation operation, the terminal device 110 runs an application 120 with a question-and-answer function, captures image data, and indicates questions about the image data.

[0045] The terminal device 110 may capture image data using an image acquisition device. The image data may be in any form (e.g., a still image or a video clip), at any resolution, and in any format (e.g., PNG, JPG, etc.). Alternatively or additionally, the image data may be data pre-stored in the terminal device 110.

[0046] Questions can include multiple forms, that is, the terminal device 110 can capture questions in multiple forms. Questions can include, for example, questions captured in text form, and the terminal device 110 can directly obtain the text sequence of questions input by the user (for example, obtaining the text sequence of "How many cups are there in the image?"). In order to ensure the convenience and ease of operation of questions and answers, questions can also include questions captured in voice form, and the terminal device 110 can capture the voice data of the user's questions through a sound collection device. The voice data can be in any language (for example, Chinese, English, Japanese, etc.), any time length (for example, 3s, 5s, etc.) and any timbre. It can be understood that questions can also be questions captured in any other appropriate form.

[0047] In the case where the question is captured in the form of voice, in some embodiments, after the terminal device 110 launches the application 120, a question-and-answer interface including at least a recording control may be presented via a display device. In response to detecting a predetermined operation on the recording control, the terminal device 110 may determine that a question-and-answer initiation operation has been detected, and then capture image data and voice data. The predetermined operation on the recording control may include, for example, a click operation, a slide operation, a long press operation, etc., without limitation herein. In some embodiments, the predetermined operation on the recording control may also be initiated via voice or other instructions.

[0048] In some embodiments, during the process of capturing image data and voice data, the terminal device 110 may also stop capturing image data and voice data in response to receiving a capture end operation. Specifically, the terminal device 110 may determine that a capture end operation has been detected in response to detecting a voice such as "stop collecting data". The terminal device 110 may also determine that a capture end operation has been detected in response to detecting a preset operation on a hardware button (such as a press operation, a long press operation, etc.). The terminal device 110 may also determine that a capture end operation has been detected in response to detecting another predetermined operation on a recording control in the recording interface (such as a click operation, a release press operation, etc.).

[0049] Referring to Figure 3, a schematic diagram of a Q&A interface 300 according to some embodiments of the present disclosure is shown. Recording interface 300 may include a control display area 330, which displays at least a recording control 332. Terminal device 110 may determine that a Q&A initiation operation has been detected in response to detecting a predetermined operation on recording control 332. Terminal device 110 may then display text such as "Recording" in text prompt area 310 to indicate to the user that terminal device 110 is currently capturing data.

[0050] In some embodiments, in response to receiving a predetermined operation, the terminal device 110 changes the presentation of the recording control 332 (e.g., by changing the color or size of the recording control 332) to indicate that the terminal device 110 is capturing voice data and image data. Accordingly, in response to receiving a capture end operation, the terminal device 110 can stop capturing voice data and image data. In response to receiving a capture end operation, the terminal device 110 can switch the presentation of the recording control 332 back to the state before capture.

[0051] In some embodiments, terminal device 110 can convert captured voice data into text and display it in text prompt area 310. As shown in Figure 3, after terminal device 110 captures voice data, it displays the text corresponding to the voice data, "Dosage of this medicine," in text prompt area 310. Simultaneously, terminal device 110 can present the image data currently being captured by terminal device 110 in image display area 320. Speech-to-text conversion can be achieved using speech-to-text technology, which can be performed locally on terminal device 110 or on a remote server.

[0052] In some embodiments, to ensure the accuracy of the question indicated by the determined voice data, the terminal device 110 may pre-process the captured voice data to eliminate noise (such as ambient sound) in the voice data that is not related to the question.

[0053] Referring back to FIG. 2 , at block 220 , the terminal device 110 extracts text information from the image data.

[0054] The terminal device 110 can, for example, use optical character recognition (OCR) technology to extract text information from image data. Optical character recognition refers to the process of analyzing, identifying, and processing image data to obtain text information. OCR technology allows image data to undergo image processing processes such as image preprocessing, text detection, and text recognition to extract text information from image data. Specifically, in OCR technology, image preprocessing is usually to correct imaging problems of image data. Common preprocessing processes include: geometric transformation (perspective, distortion, rotation, etc.), distortion correction, blur removal, image enhancement, and light correction. Text detection is to detect the location and range of the text and its layout. It usually also includes layout analysis and text line detection. Text recognition is to identify the text content on the basis of text detection and extract text information from image data.

[0055] In some embodiments, the terminal device 110 can extract text information from image data using a trained image processing model based on OCR technology. Specifically, the terminal device 110 can input the image data into the trained image processing model and obtain the text information output by the image processing model. The image processing model can include, for example, one or more of any appropriate models such as a convolutional neural network (CNN), a feedforward neural network (FNN), a fully connected neural network (FCN), a generative adversarial network (GAN), and a recurrent neural network (RNN).

[0056] In some embodiments, the image processing model can be deployed in the server 130, for example, wherein the server 130 can be a remote server (e.g., a cloud server). The terminal device 110 can utilize the trained image processing model to obtain text information through communication with the server 130. Specifically, the terminal device 110 can send the captured image data to the server 130, and the server 130 can output text information based on the image data using the trained image processing model. The terminal device 110 can obtain text information from the server 130. In some embodiments, the trained image processing model can also be deployed locally on the terminal device 110, for example, and the terminal device 110 can directly utilize the trained image processing model deployed locally to extract text information based on the captured image data.

[0057] In block 230 , the terminal device 110 obtains extended information associated with the text information.

[0058] In an embodiment of the present disclosure, in a scenario where a user desires to ask questions about image data, it is not only necessary to rely on the image data itself, but also to expand more information to assist in answering the questions.

[0059] In some embodiments, the terminal device 110 may always obtain extended information associated with the text information in the image data by default. In some embodiments, the terminal device 110 may determine whether a target answer to a question can be determined from the image data. Specifically, the terminal device 110 may perform intent recognition on the question to determine the intent associated with the question. Specifically, the terminal device 110 may identify intent based on a rule-based approach using dictionaries and templates. Different intents may have different domain dictionaries, such as book titles, song titles, product names, object names, etc. The terminal device 110 may make a determination based on the degree of match or overlap between the user's intent and the dictionary. The terminal device 110 may also identify user intent based on a machine learning model. The terminal device 110 may train and learn from annotated domain corpus using machine learning and deep learning methods to obtain an intent recognition model (e.g., a model based on fastText). The terminal device 110 then identifies the intent indicated by the input question based on this model. It is understood that the terminal device 110 may determine the intent indicated by the question locally or send the question to the server 130 for server 130 to determine the intent indicated by the question.

[0060] After the terminal device 110 obtains the intention of the question indication, it can determine whether the target answer to the question can be determined from the image data based on the intention and the image data and / or text information.

[0061] If, based on the intent and the image data, it is determined that the target answer to the question can be determined from the image data, the terminal device 110 can directly determine the corresponding target answer based on the image data and the question without obtaining extended information. For example, if the terminal device 110 captures voice data corresponding to the question "How many cups are there?", the terminal device 110 can recognize the voice data and determine that the intent corresponding to the question is to determine the number of cups in the image data. The terminal device 110 then determines that the target answer to the question can be determined from the image data based on the intent and the captured image data. The terminal device 110 can directly recognize the image data and determine that the number of cups included in the image data is 3, that is, the answer corresponding to the intent is "3".

[0062] For another example, if the question captured by terminal device 110 is voice data corresponding to "What is the dosage of a drug?", terminal device 110 may recognize the voice data and determine that the question is intended to determine the dosage of the drug in the image data. If the text on the drug packaging in the image data is too small, the dosage is not included on the drug packaging, or the text on the drug packaging is blurred, terminal device 110 may determine that the target answer to the question cannot be determined from the image data based on the intention and the text information extracted from the captured image data. If it is determined that the target answer to the question cannot be determined from the image data, terminal device 110 may obtain extended information associated with the text information.

[0063] In some embodiments, the terminal device 110 can access a knowledge base and obtain extended information from the knowledge base based on text information. The knowledge base may include a knowledge graph. Specifically, the terminal device 110 can determine at least one keyword from the text information and obtain extended information associated with at least one keyword from the knowledge base. The terminal device 110 can match keywords to entities, relationships, attributes, etc. in the knowledge graph to obtain corresponding information as extended information. Knowledge graph is an important data representation form in computer science. In the knowledge graph, nodes represent entities, and edges between nodes represent relationships between entities. Both nodes and edges can have their own attributes, which are attributes of the entity or relationship. It can be understood that the knowledge base can also be any other suitable knowledge base and / or database, etc., which is not limited here.

[0064] In order to obtain more accurate extended information, the knowledge base pre-stored by the terminal device 110 may include multiple candidate knowledge bases corresponding to multiple fields. The multiple candidate knowledge bases include, but are not limited to, a medical knowledge base corresponding to the medical field, an agronomic knowledge base corresponding to agriculture, a pet knowledge base corresponding to pets, a food knowledge base corresponding to food, and the like. The terminal device 110 can determine the target field corresponding to the image data or text information, and then determine the target knowledge base corresponding to the target field from the multiple candidate knowledge bases. For example, if the text information of the image data contains the name of a medicine, the terminal device 110 can determine that the field corresponding to the text information is the medical field, and then determine the medical knowledge base corresponding to the medical field as the target knowledge base from the multiple candidate knowledge bases.

[0065] Furthermore, the terminal device 110 may obtain extended information associated with at least one keyword from the target knowledge base. Similarly, if the target knowledge base is a knowledge graph, the terminal device 110 may match at least one keyword with entities, relationships, attributes, etc. in the knowledge graph to obtain corresponding information as extended information.

[0066] In some embodiments, terminal device 110 may also obtain extended information based on the image data. Specifically, terminal device 110 may perform image recognition on the image data and, based on the image recognition results, obtain extended information from a target knowledge base in the field corresponding to the image data. For example, if the image recognition results indicate that the image data contains a cat, terminal device 110 may determine a pet knowledge base corresponding to the pet as a target knowledge base from among multiple candidate knowledge bases, and then obtain extended information associated with the cat from the target knowledge base.

[0067] In block 240 , the terminal device 110 determines a target answer to the question based on the image data and the extended information.

[0068] In some embodiments, the terminal device 110 can locally use the image data and extended information to determine the answer to the question. Specifically, the terminal device 110 can, for example, use a trained question-answering model to determine the answer to the image data based on the image data, extended information, and the question. In some embodiments, when the question is captured in the form of voice, the terminal device 110 can first convert the voice data corresponding to the question into a text sequence. The terminal device 110 can convert the voice data into a text sequence based on voice technology (for example, automatic speech recognition (ASR) technology), and then provide the image data, extended information, and text sequence to the question-answering model.

[0069] In some embodiments, the terminal device 110 can input the text sequence, image data, and extended information corresponding to the question into a trained question-answering model, so that the question-answering model outputs an answer corresponding to the question. In this case, the question-answering model is a multimodal question-answering model, and its input includes data in image modality and text modality. Alternatively or additionally, in some embodiments, the terminal device 110 can also input the text sequence, text information extracted from the image data, and extended information into a trained question-answering model, so that the question-answering model outputs an answer corresponding to the question. In this case, the question-answering model may not be a multimodal question-answering model, and its input is data in text modality.

[0070] It is understood that the trained question-answering model can be deployed on the server 130 or locally. If the question-answering model is deployed on the server, the terminal device 110 can utilize the trained question-answering model to implement the question-answering function through communication with the server 130. If the question-answering model is deployed locally, the terminal device 110 can directly utilize the trained question-answering model deployed locally to implement the question-answering function.

[0071] The question-answering model can be, for example, a language model (LM). A language model can acquire question-answering capabilities by learning from a large amount of corpus. Language models can include statistical language models and neural network language models. Compared to statistical language models, neural network language models have stronger generalization and prediction capabilities. In some embodiments, to better implement the question-answering function, the trained question-answering model used is a neural network language model.

[0072] Furthermore, since the question-answering capability of the neural network language model can be improved as the amount of data used for training and the amount of model parameters increase, in some embodiments, in order to determine a more accurate answer, the terminal device 110 uses a neural network language model with large-scale parameters, data volume, and computing power to meet the question-answering quality requirements in specific applications. When the scale of the language model reaches a certain level (for example, trained by a larger amount of data), it will have cognitive, common sense, and logical reasoning capabilities that meet the application expectations. In some embodiments, to ensure the effectiveness of the model, the parameter weights of the model can be determined by pre-training.

[0073] In some embodiments, the terminal device 110 can also determine candidate answers to the question from the image data, and perform corrections on the candidate answers based on the extended information to obtain a target answer. Correction may, for example, include at least one of error correction and completion. Specifically, the terminal device 110 can, for example, use a trained question-answering model to determine candidate answers to the image data based on the image data and the question. Similarly, in the case where the question is captured in the form of voice, the terminal device 110 needs to first convert the voice data corresponding to the question into a text sequence, and then input the text sequence into the trained question-answering model together with the image data, so that the question-answering model outputs the candidate answer corresponding to the question. Correction of candidate answers based on image data (for example, error correction and / or completion) can be automatically implemented by the question-answering model.

[0074] If the image data quality is poor, the candidate answers determined directly by the terminal device 110 based on the image data may be incorrect. Therefore, the terminal device 110 can further correct the candidate answers based on the extended information to obtain an accurate target answer to the question. The terminal device 110 can correct the candidate answers based on the information corresponding to the question in the extended information to obtain the correct target answer.

[0075] For example, when the question is "name and function of the drug", if the image data has an improper exposure problem, the text area on the drug packaging in the image data may be unclear. The candidate answer obtained by the terminal device 110 based on such image data may be incomplete. For example, the candidate answer may be "Huanglian Shangqing Tablets, which can dispel wind and clear heat." After the terminal device 110 uses the extended information to complete the candidate answer, the target answer can be "Huanglian Shangqing Tablets, which can dispel wind and clear heat, purge fire and relieve pain." Therefore, the extended information can be used to perform a complete correction on the candidate answer, which helps to improve the completeness of the answer.

[0076] For example, if the question is "What is the dosage of this medicine?", if the text on the drug packaging in the image data does not contain the corresponding content or the corresponding text is too small, the candidate answer obtained by terminal device 110 based on such image data may be incorrect. For example, the candidate answer may be "24 tablets, 2 boxes". After using the extended information to correct the candidate answer, terminal device 110 can obtain the target answer as "6 tablets at a time, twice a day". Therefore, the extended information can be used to correct the candidate answer, helping to improve the accuracy of the answer.

[0077] Referring to FIG. 4 , FIG. 4 illustrates a schematic diagram of a response process 400 according to some embodiments of the present disclosure. Process 400 may be implemented at terminal device 110 . For ease of discussion, process 400 will be described with reference to environment 100 of FIG. 1 . Note that image data 401 , question 402 , model input 403 , and response 404 shown in FIG. 4 are merely examples for illustrative purposes and are not intended to indicate any limitation.

[0078] In some embodiments, after acquiring the image data 401, the terminal device 110 performs text recognition 410 on the image data 401 to extract text information therefrom, such as extracting text information "XX brand", "Huanglian Shangqing Tablets", "24 tablets·2 plates", "Disperse wind and clear heat, purge fire and relieve pain" and "OTC" from the image data 401. The terminal device 110 obtains extended information associated with the text information from the knowledge base 420 (for example, a medical-related knowledge graph). For example, the extended information "Effects and effects: This product has the effects of dispersing wind and clearing heat, purging fire and relieving pain. It can treat acute conjunctivitis, acute suppurative otitis media, toothache, laryngeal paralysis, as well as mouth ulcers and recurrent mouth ulcers" and "Usage and dosage: oral medication, 6 tablets at a time, twice a day. Patients are advised to use the medication under the guidance of a physician", etc. can be obtained from the knowledge 402.

[0079] In some embodiments, as shown in FIG4 , terminal device 110 may determine text information, extended information, and question 402 together as model input 403 for a question-answering model. In other examples, terminal device 110 may also determine model input 403 for a question-answering model based on image data 401, extended information, and question 402 (e.g., in the form of speech). When determining model input 403, terminal device 110 may recognize captured speech data as text and determine the intent of the question.

[0080] After determining model input 403, terminal device 110 provides model input 403 to question-answering model 430. Question-answering model 430 may determine an answer 404 corresponding to the question based on model input 403. Terminal device 110 may obtain answer 404 output by question-answering model. For example, answer 404 may be "6 tablets at a time, twice a day."

[0081] In some embodiments, after the terminal device 110 determines the answer, it can play the answer in voice form through a sound playback device. As shown in Figure 3, the terminal device 110 can play the answer audio through a speaker. In some embodiments, the answer can be in text form. The terminal device 110 can convert the text into speech for output through text-to-speech synthesis (TTS). This makes it convenient for users, especially those with visual impairments, to quickly obtain the answer.

[0082] In some embodiments, the terminal device 110 may alternatively present the answer in text form via the display screen. In some embodiments, the terminal device 110 may also output the answer in an additional vibrational or visual form. Visual forms may include, for example, magnified or highlighted images. For example, if the user inputs voice data indicating the name of an object in the query image data, the terminal device 110 may, while playing the audio answer containing the object's name, magnify the image data on the display screen to highlight the object.

[0083] According to the embodiments of the present disclosure, when poor quality captured images prevent the completion of Q&A, the proposed solution can significantly improve the efficiency and accuracy of Q&A, compared to requiring the user to recapture images that meet the requirements. In this way, a knowledge base can be introduced in multimodal data Q&A scenarios to expand the ability to accurately answer questions. This allows users to be provided with instant and accurate Q&A services even when image data is incomplete or insufficient.

[0084] Figure 5 shows a schematic structural block diagram of a question-answering apparatus 500 according to some embodiments of the present disclosure. Apparatus 500 may be implemented in or included in terminal device 110, for example. Each module / component in apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0085] As shown, apparatus 500 includes a data capture module 510 configured to, in response to detecting a question-and-answer initiation operation, capture image data and a question about the image data using a user's device. Apparatus 500 also includes a text information extraction module 520 configured to extract text information from the image data. Apparatus 500 also includes an extended information acquisition module 530 configured to acquire extended information associated with the text information. Apparatus 500 also includes a target answer determination module 540 configured to determine a target answer to the question based on the image data and the extended information.

[0086] In some embodiments, the extended information acquisition module 530 includes: a keyword determination module configured to determine at least one keyword from text information; and an extended information acquisition module configured to acquire extended information associated with the at least one keyword from a knowledge base.

[0087] In some embodiments, the knowledge base includes multiple candidate knowledge bases corresponding to multiple fields, and the extended information acquisition module includes: a target field determination module, configured to determine the target field corresponding to the image data or text information; a target knowledge base determination module, configured to determine the target knowledge base corresponding to the target field from multiple candidate knowledge bases; and an information acquisition module, configured to obtain extended information associated with at least one keyword from the target knowledge base.

[0088] In some embodiments, the knowledge base includes a knowledge graph.

[0089] In some embodiments, the extended information acquisition module 530 includes: a determination module configured to determine whether the target answer to the question can be determined from the image data; and an acquisition module configured to acquire extended information if the target answer to the question cannot be determined from the image data.

[0090] In some embodiments, the target answer determination module 540 includes: a candidate answer determination module, configured to determine candidate answers to the question from image data; and a target answer acquisition module, configured to perform correction on the candidate answers based on extended information to obtain the target answer, the correction including at least one of error correction and completion.

[0091] In some embodiments, the target answer is determined using a trained question-answering model, the model input of the question-answering model including at least one of image data and text information, extended information, and a question.

[0092] In some embodiments, the question comprises a question captured in speech form.

[0093] The units included in the device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units in the device 500 can be implemented at least in part by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0094] FIG6 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 600 shown in FIG6 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 600 shown in FIG6 can be used to implement the electronic device 110 of FIG1 .

[0095] As shown in FIG6 , electronic device 600 is a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to enhance the parallel processing capabilities of electronic device 600.

[0096] The electronic device 600 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the electronic device 600.

[0097] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG6 , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0098] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0099] The input device 650 may be one or more input devices, such as a mouse, keyboard, or trackball. The output device 660 may be one or more output devices, such as a display, a speaker, or a printer. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with the electronic device 600, or with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0100] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0101] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0102] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0103] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0104] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0105] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A question-answering method, comprising: In response to detecting the question-and-answer initiation operation, capturing image data and a question for the image data using a device of the user; extracting text information from the image data; Acquire extended information associated with the text information; as well as A target answer to the question is determined based on the image data and the extended information.

2. The method according to claim 1, wherein obtaining the extended information comprises: determining at least one keyword from the text information; as well as The extended information associated with the at least one keyword is obtained from a knowledge base.

3. The method according to claim 2, wherein the knowledge base includes a plurality of candidate knowledge bases corresponding to a plurality of fields, and obtaining the extended information from the knowledge base includes: Determining a target area corresponding to the image data or the text information; Determine a target knowledge base corresponding to the target domain from the plurality of candidate knowledge bases; as well as The extended information associated with the at least one keyword is obtained from the target knowledge base.

4. The method according to claim 2, wherein the knowledge base comprises a knowledge graph.

5. The method according to claim 1, wherein obtaining the extended information comprises: determining whether a target answer to the question can be determined from the image data; as well as If the target answer to the question cannot be determined from the image data, the extended information is acquired.

6. The method of claim 1, wherein determining a target answer to the question comprises: determining candidate responses to the question from the image data; as well as Based on the extended information, correction is performed on the candidate answer to obtain the target The correction includes at least one of error correction and completion.

7. The method according to claim 1, wherein the target answer is determined using a trained question-answering model, the model input of the question-answering model including at least one of the image data and the text information, the extended information and the question. The method of claim 1 , wherein the question comprises a question captured in speech form.

9. A device for question answering, comprising: A data capture module, configured to capture image data and questions regarding the image data using a user's device in response to detecting a question-and-answer initiation operation; A text information extraction module, configured to extract text information from the image data; An extended information acquisition module, configured to acquire extended information associated with the text information; as well as The target answer determination module is configured to determine a target answer to the question based on the image data and the extended information.

10. The device according to claim 9, wherein the extended information acquisition module comprises: A keyword determination module, configured to determine at least one keyword from the text information; as well as The extended information acquisition module is configured to acquire the extended information associated with the at least one keyword from a knowledge base.

11. The apparatus according to claim 10, wherein the knowledge base comprises a plurality of candidate knowledge bases corresponding to a plurality of fields, and the extended information acquisition module comprises: a target domain determination module, configured to determine a target domain corresponding to the image data or the text information; a target knowledge base determination module, configured to determine a target knowledge base corresponding to the target domain from the plurality of candidate knowledge bases; as well as The information acquisition module is configured to acquire the extended information associated with the at least one keyword from the target knowledge base.

12. The apparatus according to claim 10, wherein the knowledge base comprises a knowledge graph.

13. The device according to claim 9, wherein the extended information acquisition module comprises: a determination module configured to determine whether a target answer to the question can be determined from the image data; as well as The acquisition module is configured to acquire the extended information if the target answer to the question cannot be determined from the image data.

14. The apparatus according to claim 9, wherein the target answer determination module comprises: a candidate answer determination module configured to determine a candidate answer to the question from the image data; as well as The target answer obtaining module is configured to perform correction on the candidate answer based on the extended information to obtain the target answer, wherein the correction includes at least one of error correction and completion.

15. The apparatus according to claim 9, wherein the target answer is determined using a trained question-answering model, a model input of the question-answering model comprising at least one of the image data and the text information, the extended information, and the question.

16. The apparatus of claim 9, wherein the question comprises a question captured in speech form.

17. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 8 when executed by the at least one processing unit.

18. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 8 when executed by a processor.

19. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Visual question and answer processing method and device, computer readable medium and program product

    CN113722458A

  • Visual question-answering method and device, and medium

    CN113780486A

  • Question answering system and method, electronic equipment and storage medium

    CN114911910A

  • Visual question and answer method and device and storage medium

    CN115618045A

  • Question and answer method and device, equipment and storage medium

    CN116628150A

Cited By

  • Question and answer information processing method and device based on permission tag and electronic equipment

    CN119128047A

  • Question answering method and device and computer readable storage medium

    CN119166792A

  • User question response method and system, computer equipment and readable storage medium

    CN119202142A

  • Method for providing medical question and answer service, related device and computer program product

    CN119560183A

  • Image searching method and device, intelligent agent, electronic equipment and storage medium

    CN119597954A