Interactive question and answer method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides an interactive question and answer method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a question instruction and an image containing a directional object; associating a referring word in the question instruction with an object indicated by the directive object to obtain a referring object of the referring word, and generating an answer text based on the referring object; and performing interactive question answering based on the answer text. According to the interactive questioning and answering method and device, the electronic equipment and the storage medium, the anaphora in the questioning instruction is associated with the specific object indicated by the directive object, the anaphora question of the user can be finely recognized and understood, the intention of the user is correctly understood, and the user experience is improved by intelligently analyzing the anaphora. Alignment of multi-modal information is ensured, so that answer content is closer to natural logic of human dialogues, and interaction fluency and user experience are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an interactive question-answering method, device, electronic device and storage medium. Background Art

[0002] In today's digital age, human-computer interactive question-answering systems have become an important tool for obtaining information and solving problems, and are widely used in education, medical care, customer service, smart home, entertainment, etc. These systems use natural language processing technology to identify the user's intentions and provide accurate and timely answers or perform corresponding operations.

[0003] However, natural language itself is complex and diverse, and the way users ask questions may vary depending on cultural background, education level, personal habits, etc., making it difficult for the system to accurately capture the user's true intentions. For example, users may use colloquial expressions, referential expressions, etc. In actual operation, the relevant question-and-answer system often faces the problem of misunderstanding when identifying user intentions and interacting, which has a significant impact on the user's interactive experience. Summary of the invention

[0004] The present invention provides an interactive question-answering method, device, electronic device and storage medium, which are used to solve the defect in the prior art that when identifying user intentions and interacting, the understanding deviation affects the user interaction experience.

[0005] The present invention provides an interactive question-answering method, comprising: Obtain questioning instructions and images containing pointed objects; Associating the pronoun in the question instruction with the object indicated by the directional object to obtain the referent of the pronoun, and generating an answer text based on the referent; Based on the answer text, interactive question and answer is performed.

[0006] According to the interactive question-answering method provided by the present invention, associating the pronoun in the question instruction with the object indicated by the directional object to obtain the object referred to by the pronoun includes: Performing object detection on the image to obtain the directional object and the position of each object; Based on the directional object and the positions of each object, the pronoun in the question instruction is associated with the object indicated by the directional object to obtain the object referred to by the pronoun.

[0007] According to the interactive question-answering method provided by the present invention, the pronoun in the question instruction is associated with the object indicated by the directional object based on the directional object and the position of each object to obtain the object referred to by the pronoun, including: Based on the positions of the directional object and each object, determining the distance between the directional object and each object, and the indication direction of the directional object; Based on the distance and the indicated direction, the pronoun in the question instruction is associated with the object indicated by the directional object to obtain the object referred to by the pronoun.

[0008] According to the interactive question-answering method provided by the present invention, the interactive question-answering based on the answer text includes: Based on the question instruction, predicting the emotional state contained in the answer text; Perform speech synthesis based on the answer text and the emotional state to obtain an answer speech; Based on the answer voice, interactive question and answer is performed.

[0009] According to the interactive question-answering method provided by the present invention, the method further comprises: When the current question instruction does not contain a pronoun, or the referent of the pronoun in the current question instruction is known based on historical questions and answers, interactive questions and answers are performed based on the large voice model, the current question instruction and the current image.

[0010] According to the interactive question-answering method provided by the present invention, associating the pronoun in the question instruction with the object indicated by the directional object to obtain the referent of the pronoun, and generating an answer text based on the referent, includes: Based on a large multimodal understanding model, the pronoun in the question instruction is associated with the object indicated by the directional object to obtain the referent of the pronoun, and a response text is generated based on the referent. The large multimodal understanding model is obtained by training a large language model in stages based on sample question instructions and sample images.

[0011] According to the interactive question-and-answer method provided by the present invention, the question instruction includes a text instruction or a voice instruction, and the image includes a single image, multiple images containing dynamic directional objects in a single scene, multiple images containing dynamic directional objects across scenes, and multiple video frames with semantic associations.

[0012] The present invention also provides an interactive question-answering device, comprising: An acquisition unit, used for acquiring a question instruction and an image containing a directional object; A generating unit, configured to associate the pronoun in the question instruction with the object indicated by the directional object, obtain the object referred to by the pronoun, and generate an answer text based on the object referred to; The interactive unit is used for conducting interactive question and answer based on the answer text.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned interactive question-and-answer methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the interactive question-and-answer method as described above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the interactive question-and-answer method described above is implemented.

[0016] The interactive question-and-answer method, device, electronic device, and storage medium provided by the present invention can finely identify and understand the user's referential questions by associating the pronouns in the question instructions with the specific objects indicated by the directional objects, thereby correctly understanding the user's intentions, and ensuring the alignment of multimodal information through intelligent parsing of pronouns, so that the answer content is closer to the natural logic of human dialogue, thereby improving the fluency of interaction and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the flow charts of the interactive question-answering method provided by the present invention.

[0019] Figure 2 This is one of the flowchart diagrams of step 120 in the interactive question-answering method provided by the present invention.

[0020] Figure 3 This is the second flowchart diagram of step 120 in the interactive question-and-answer method provided by the present invention.

[0021] Figure 4 It is one of the image schematic diagrams provided by the present invention.

[0022] Figure 5This is the second image schematic diagram provided by the present invention.

[0023] Figure 6 This is the third image schematic diagram provided by the present invention.

[0024] Figure 7 This is the fourth image schematic diagram provided by the present invention.

[0025] Figure 8 It is a structural schematic diagram of the interactive question-answering device provided by the present invention.

[0026] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] In order to solve the problem that the understanding deviation affects the user interaction experience when identifying the user intention and interacting, an embodiment of the present invention proposes an interactive question-and-answer method. In this method, firstly, a question instruction and an image containing a directional object are obtained; the pronoun in the question instruction is associated with the object indicated by the directional object to obtain the referent of the pronoun, and an answer text is generated based on the referent; then, based on the answer text, an interactive question-and-answer is performed. By associating the pronoun in the question instruction with the specific object indicated by the directional object, the user's referential questions can be finely identified and understood, so as to correctly understand the user's intention. By intelligently parsing the pronoun, the alignment of multimodal information is ensured, so that the answer content is closer to the natural logic of human dialogue, and the fluency of interaction and user experience are improved.

[0029] The embodiments of the present invention can be applied to scenarios where interactive question-answering is required, especially scenarios where directional interactive question-answering capabilities are required. The execution subject of the method can be an electronic device such as a terminal device, a computer, a server, a server cluster, or a specially designed interactive question-answering device, or an interactive question-answering device set in the electronic device, and the interactive question-answering device can be implemented by software, hardware, or a combination of the two. The terminal device can be, for example, a learning machine, a smart wearable device, etc.

[0030] Figure 1 It is one of the flow charts of the interactive question-answering method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 110: Obtain a question instruction and an image containing a directional object.

[0031] Specifically, the question instruction may be an instruction raised by the user in the form of natural language, which requires a targeted response or answer. The question instruction may be a question raised by the user, or an instruction issued by the user, or may include both a question and an instruction, which is not specifically limited in the embodiment of the present invention. For example, the question instruction may be "Excuse me, what is this word?", "Read these lines of words and see if I can eat if my throat hurts?".

[0032] The question instruction may be text data or voice data. For example, the user may input text data through a keyboard, a touch screen or other device, or may input voice data through a microphone or other sound pickup device.

[0033] Here, the image containing the directional object may be pre-stored or captured in real time, and the embodiment of the present invention does not specifically limit this. For example, the user inputs a pre-stored image while inputting a question instruction, or the question-answering device automatically starts a camera or captures the image in response to the user's operation when receiving the question instruction. The image may be a static image or a video frame.

[0034] The directional objects contained in the image may be, for example, fingers, pens, pointers, arrows, etc. The directional objects are used to indicate one or more object entities in the image, such as people, text, animals, buildings, dolls, plants, equipment, etc. in the image.

[0035] Step 120, associating the pronoun in the question instruction with the object indicated by the directional object to obtain the referent of the pronoun, and generating an answer text based on the referent; Step 130, conducting interactive question and answer based on the answer text.

[0036] Specifically, pronouns are words used to replace nouns or noun phrases to avoid repetition while keeping sentences fluent and concise, and are often used in natural language instructions. Pronouns here usually include demonstrative pronouns, such as "this", "that", "these", "those", "here", etc.

[0037] Considering that when the question instruction contains pronouns, the question-answering device has deviations in its intention recognition and understanding of the question instruction containing pronouns, or cannot accurately identify the content referred to by the pronouns, or cannot understand the question instruction, resulting in the inability to conduct directed interactive question-answering, affecting the user interaction experience.

[0038] In this embodiment, by associating the pronouns in the question instruction with the objects indicated by the directive objects, the referents of the pronouns are obtained, thereby enabling accurate conversion of the question instruction. For example, the question instruction is "What is this character?", and by associating the pronoun "this" in the question instruction with the character indicated by the directive object (finger) in the image, the referent of the pronoun "this" is obtained as the Chinese character "䧿".

[0039] Subsequently, an answer text can be generated based on the referent "䧿 mountain". For example, the answer text generated for the question instruction "What is this character?" can be "This character is pronounced què", and further, an explanation and origin of the character "䧿" can also be generated.

[0040] Regarding associating the pronouns in the question instruction with the objects indicated by the directive objects to obtain the referents of the pronouns, it can be achieved by analyzing the spatial relationship between the directive object and the objects in the image. If the spatial association relationship between the directive object and the object is greater, then the possibility of the object being the referent of the pronoun is greater; conversely, if the spatial association relationship between the directive object and the object is smaller, then the possibility of the object being the referent of the pronoun is smaller. After obtaining the referent, according to a further understanding of the question instruction, an information retrieval algorithm is used to find the answer most relevant to the question instruction, or natural language technology is used to generate an answer text for the question instruction.

[0041] In some other embodiments, the referents of the pronouns and the generation of answer texts can be determined by a trained multi-modal understanding large model. The multi-modal understanding large model is a multi-modal large model, which refers to a model that can process and analyze data information from different sources and different forms. These data information may include various types such as text, image, audio, video, etc. The question instruction and the image containing the directive object are input into the multi-modal understanding large model, and the multi-modal understanding large model comprehensively analyzes and understands the question instruction and the image, locates the pronouns in the question instruction to specific objects in the image, obtains the referents of the pronouns, and generates an answer text based on the referents.

[0042] On this basis, step 130 is executed to perform interactive Q&A based on the answer text.

[0043] After obtaining the answer text, the answer text can be directly output as the answer, or the answer text can be synthesized into speech and the synthesized speech can be output as the answer.

[0044] Users can conduct further interactions based on the interactive answers. For example, if the user needs further explanations or clarifications, or further precise positioning, or to expand the Q&A based on a certain object, the system will analyze in combination with the current image, question and historical Q&A, and provide additional information or conduct further conversations.

[0045] The method provided by the embodiment of the present invention can finely identify and understand the user's referential questions by associating the pronouns in the question instructions with the specific objects indicated by the directional objects, thereby correctly understanding the user's intentions, and ensuring the alignment of multimodal information by intelligently parsing the pronouns, so that the answer content is closer to the natural logic of human dialogue, thereby improving the fluency of interaction and user experience.

[0046] Based on the above embodiments, Figure 2 is one of the flow charts of step 120 in the interactive question-answering method provided by the present invention, such as Figure 2 As shown, the pronoun in the question instruction is associated with the object indicated by the directional object to obtain the object referred to by the pronoun, that is, step 120 specifically includes: Step 121, performing object detection on the image to obtain directional objects and the positions of each object; Step 122 : Based on the directional object and the position of each object, the pronoun in the question instruction is associated with the object indicated by the directional object to obtain the object referred to by the pronoun.

[0047] Specifically, directional objects and the positions of each object can be achieved by performing object detection on the image through an object detection model. The object detection model can identify multiple objects in the image and obtain the category label and location information (bounding box) of each object. The identified objects may include directional objects and other objects. Determine which object is a "directional object" based on specific rules or contextual information. Directional objects are usually objects that indicate the user's intention to point or focus on, such as the fingers of people in the image, arrows, light beams, indicator sticks, etc. Other objects may include, for example, people, text, animals, buildings, dolls, plants, equipment, etc.

[0048] On this basis, analyze the pointing direction or focus range of the directional object according to the identified directional objects and their positions. Traverse all identified objects and associate them with the pointing direction or focus range of the directional object according to their position information. Find the object located in the pointing direction or focus range of the directional object. Match the pronoun in the question instruction with the associated object to determine the referent of the pronoun.

[0049] Based on the above embodiments, Figure 3 is a second flow chart of step 120 in the interactive question-answering method provided by the present invention, such as Figure 3 As shown, based on the directional objects and the positions of the objects, the pronouns in the question instruction are associated with the objects indicated by the directional objects to obtain the objects referred to by the pronouns, that is, step 122 specifically includes: Step 122-1, based on the positions of the directional object and each object, determining the distance between the directional object and each object, and the indication direction of the directional object; Step 122 - 2 , based on the distance and the indicated direction, the pronoun in the question instruction is associated with the object indicated by the directional object to obtain the object referred to by the pronoun.

[0050] Specifically, associating the pronoun in the question instruction with the object indicated by the directional object can be achieved based on the distance between the directional object and each object, and the indicated direction of the directional object. Usually, the object with the closest distance to the directional object and the entity object included in the question instruction can be used as the pronoun's pronoun, or the object in the indicated direction of the directional object can be used as the pronoun's pronoun.

[0051] First, the position information of each object (including directional objects) in the image (usually the coordinates of the upper left and lower right corners of the bounding box) is converted into coordinate points on a two-dimensional plane. This can be achieved through geometric transformation, such as converting pixel coordinates into normalized coordinates.

[0052] For a directional object, determine the coordinates of its center point or a key point (e.g., finger tip, arrow tip, etc.). Calculate the distance between the center point or key point of the directional object and the center point of each other object. This can be achieved using the Euclidean distance formula.

[0053] The pointing direction of the directional object is determined based on its shape, direction, or context information. For example, if the directional object is an arrow, the direction of the arrow is the pointing direction; if it is a finger, the direction can be determined by the degree of bending of the finger and the direction of the fingertip. The pointing direction can be represented as a vector on a two-dimensional plane or by an angle.

[0054] According to the calculated distance, a threshold is set to filter out objects that are closer to the directional object as candidate objects. This threshold can be adjusted according to the actual situation to ensure that only objects that may be pointed to are included. For each candidate object, the vector (or angle) from the directional object to the object is calculated. This vector (or angle) is compared with the indicated direction of the directional object to determine whether they match. The degree of match can be measured by the dot product of the vectors, the cosine value of the angle, or other similarity measures. Among all candidate objects, the object with the closest distance and the highest direction match is selected as the referent of the pronoun. If there are multiple objects that meet the conditions, further screening can be performed based on other factors (such as the size and significance of the object).

[0055] The method provided in the embodiment of the present invention associates the pronoun in the question instruction with the object indicated by the directional object based on the distance between the directional object and each object, and the indicated direction of the directional object, to obtain the reference object of the pronoun, which can further improve the accuracy of identifying pronoun questions, thereby accurately understanding the user's intention and improving the user's interactive experience.

[0056] Based on any of the above embodiments, Figure 3 is a flow chart of step 130 in the interactive question-answering method provided by the present invention, such as Figure 3 As shown, interactive question and answer is performed based on the answer text, that is, step 130 specifically includes: Step 131, predicting the emotional state contained in the answer text based on the question instruction; Step 132, performing speech synthesis based on the answer text and the emotional state to obtain an answer speech; Step 133, performing interactive question and answer based on the answering voice.

[0057] Specifically, in order to further improve the interactive experience, after generating the answer text, the situation state contained in the answer text can be further predicted based on the question instruction. The emotional state of the answer text can be predicted based on the emotion (such as happiness, humor, sadness, neutrality, etc.) contained in the text or voice corresponding to the question instruction. The question instruction and the answer text can be input into the trained emotion classification model to obtain the predicted emotional state.

[0058] According to the predicted emotional state, the parameters of the speech synthesis system are adjusted to reflect the corresponding emotion in the speech. This may involve adjusting the speaking speed, intonation, volume, or adding specific emotional words and phrases. The answer text is then input into the adjusted speech synthesis system to generate an answer speech containing the predicted emotion. The generated answer speech is output to the user, completing an interactive question and answer session.

[0059] The method provided by the embodiment of the present invention can further improve the interactive experience by performing speech synthesis based on the predicted emotional state in the answer text and performing interactive question and answer based on the synthesized answer speech.

[0060] Based on any of the above embodiments, the interactive question-answering method further includes: When the current question instruction does not contain a pronoun, or the referent of the pronoun in the current question instruction is known based on historical questions and answers, interactive questions and answers are performed based on the large voice model, the current question instruction and the current image.

[0061] Specifically, in the interactive question and answer round, if the question instruction of the current round does not contain a pronoun, or the referent of the pronoun in the current question instruction is known based on historical question and answer, for example, the question instruction of the previous round is "What fruit is this?", the answer text is "This is an apple", and the current question instruction is "What color is it?", then combined with the following text, it can be seen that the pronoun "it" in the current question instruction clearly refers to "apple".

[0062] That is, in the current round of interactive question and answer, the pronouns in the question instructions can be accurately understood, and interactive question and answer can be carried out directly based on the large voice model, question instructions and images.

[0063] The speech big model can be a big model that has been trained with a large amount of data and has strong language understanding and generation capabilities, such as iFlytek Spark, Baidu Wenxin Yiyan, etc. The speech big model here can support functions such as multi-round dialogue, context understanding, and sentiment analysis.

[0064] The current question instruction and the current image can be input into the speech big model. While processing the current question instruction, the speech big model determines whether the user's question can be satisfied by the speech big model alone. If not, it is necessary to further identify and associate the pronouns in the current question instruction to determine its referent object.

[0065] The method provided by the embodiment of the present invention can further reduce the delay and improve the interactive response speed, thereby improving the user interaction experience, by judging whether the pronoun in the current question instruction is known, and determining that in the known case, interactive question and answer is performed based on the large voice model.

[0066] Based on any of the above embodiments, associating the pronoun in the question instruction with the object indicated by the directional object to obtain the referent of the pronoun, and generating an answer text based on the referent, including: Based on the multimodal understanding large model, the pronouns in the question instructions are associated with the objects indicated by the directional objects to obtain the referents of the pronouns, and the answer text is generated based on the referents. The multimodal understanding large model is obtained by training a large language model in stages based on sample question instructions and sample images.

[0067] Specifically, associating the pronouns in the question instructions with the objects indicated by the directional objects, obtaining the pronouns' referents, and generating the answer text based on the pronouns' referents can be achieved through a trained multimodal understanding model. A multimodal model refers to a model that can process and analyze data information from different sources and in different forms. These data information may include text, images, audio, video and other types. The question instructions and the image containing the directional object are input into the multimodal understanding model, which conducts a comprehensive analysis and understanding of the question instructions and the image, locates the pronouns in the question instructions to the specific objects in the image, obtains the pronouns' referents, and generates the answer text based on the referents.

[0068] The multimodal understanding model includes a visual module, an alignment module, and a text generation module. The visual module uses a common visual encoder such as VIT to encode image features; the alignment module uses a Q-former or ReSampler module to align visual and text features, that is, to align and associate the pronouns in the question instructions with the objects indicated by the directional objects; the text generation module can autoregressively predict text sequences and generate answer text.

[0069] The multi-modal understanding model is obtained by training the large language model in stages, which can be divided into the pre-training stage and the instruction fine-tuning stage. For example, it can include 3-4 stages of training. With each stage, the number of models gradually decreases, and the model quality improves from coarse to fine.

[0070] Pre-training stage: Enhance the model's spatial orientation alignment capability and construct a large amount of orientation-referenced question-and-answer data for real scenes, document scenes, and synthetic data.

[0071] Fine-tuning stage: Construct question-answering data containing finger pointing or quasi-finger pointing with emotional connotations (including single-image and continuous multi-image data).

[0072] The training data of the multimodal understanding model includes text and image data (multiple image inputs) and speech data (excluding images), with images represented by placeholders. In addition to predicting the text, the emotion and strength of the answer are also predicted for speech synthesis.

[0073] The method provided in the embodiment of the present invention realizes interactive question and answer through a trained multi-modal understanding large model, which can reduce latency and improve interaction efficiency and accuracy.

[0074] Based on any of the above embodiments, the question instructions include text instructions or voice instructions, and the images include a single image, multiple images containing dynamic directional objects in a single scene, multiple images containing dynamic directional objects across scenes, and multiple video frames with semantic associations.

[0075] Specifically, the question instruction includes a text instruction or a voice instruction. The user can input the text instruction through a keyboard, a touch screen or other device, or input the voice instruction through a microphone or other sound pickup device.

[0076] The interactive question-and-answer method provided by the embodiment of the present invention can support single-scene static pointing question-and-answer, single-scene dynamic pointing question-and-answer, cross-scene dynamic pointing question-and-answer, and dynamic object pointing question-and-answer.

[0077] Among them, single-scene static pointing question and answer refers to static pointing question and answer based on a single scene. In this case, the image can be a single image, and the directional object can point to a specific entity or text in the image. Figure 4 It is one of the image diagrams provided by the present invention. The user's question instruction may be, for example: What is the word on this (pointing to a traffic sign with words), where is this (pointing to a certain building), I want to go to xxx place, am I going this way? (Based on the positioning, the map scene information is embedded into the model, and the user is assisted in understanding the environment and reaching the destination by pointing and asking questions).

[0078] Dynamic pointing question answering refers to dynamic pointing question answering based on a single scene, which is suitable for situations where single-point pointing is difficult to indicate. In this case, the image can be multiple images containing dynamic pointing objects in a single scene, and the change range between the multiple images is relatively small. The pointing object changes dynamically and points to multiple objects. Figure 5 This is the second image schematic diagram provided by the present invention, such as Figure 5 As shown in the figure, the user's question instruction can be, for example, "Help me read this paragraph (slide your finger, the direction changes constantly), and the model can respond in time and read this paragraph aloud. For example, the question instruction can also be: "What are these people doing?" (Slide your finger, it will point to several people), and the answer text can be: Look, the leftmost side of the direction you are pointing to is xxx, and the one next to it is xxx.

[0079] Cross-scene dynamic pointing question answering is applicable to objects with pointing in multiple scenes. In this case, the input image can be multiple images containing dynamic pointing objects in cross-scenes. Figure 6 This is the third image schematic diagram provided by the present invention, such as Figure 6 As shown in the figure, the variation between multiple images is large, but they have a certain correlation. The multimodal understanding model understands the objects pointed to in multiple scenes, and the model combines the information together to answer. For example, the question instruction can also be: What is the difference between this (entity A in scene 1), this (entity B in scene 2), and this (entity C in scene 3), which one do you recommend to me (such as the recommendation introduction of several products).

[0080] Dynamic object pointing questions and answers are applicable to the reference to dynamic objects in the scene, that is, the object pointed by the directional object is dynamic, such as a running puppy, a walking vehicle, a dancing actor, etc. Figure 7 This is the fourth image schematic diagram provided by the present invention, such as Figure 7 As shown, the images can be multiple video frames with semantic association. Having semantic association means that the image contents in the multiple video frames are associated, such as the same puppy is included in multiple video frames. For example, the question instruction can also be: What kind of dog is this (slide the finger to point to the specific dynamic object)? It runs very fast!

[0081] The method provided by the embodiment of the present invention reduces the user's cumbersome reference in a simple interactive way, strengthens perceptual understanding through reference prompts, realizes pointing in multiple ways, supports interactive Q&A, supports single-scene static pointing Q&A, single-scene dynamic pointing Q&A, cross-scene dynamic pointing Q&A, and pointing Q&A of dynamic objects, further improving the user's interactive experience.

[0082] Based on any of the above embodiments, an interactive question-answering method is provided, including: S1, obtaining a question instruction and an image containing a directional object. The question instruction includes a text instruction or a voice instruction, and the image includes a single image, multiple images containing a dynamic directional object in a single scene, multiple images containing a dynamic directional object in cross-scenes, and multiple video frames with a semantic association relationship.

[0083] S2, based on the multimodal understanding large model, associate the pronouns in the question instruction with the objects indicated by the directional objects, obtain the referents of the pronouns, and generate the answer text based on the referents. The multimodal understanding large model is obtained by training the large language model in stages based on sample question instructions and sample images. The specific steps of associating the pronouns in the question instruction with the objects indicated by the directional objects to obtain the referents of the pronouns include: Object detection is performed on the image to obtain the directional objects and the positions of each object; based on the directional objects and the positions of each object, the distance between the directional object and each object, as well as the indication direction of the directional object are determined; based on the distance and the indication direction, the pronoun in the question instruction is associated with the object indicated by the directional object to obtain the object referred to by the pronoun.

[0084] S3, based on the question instruction, predict the emotional state contained in the answer text; perform speech synthesis based on the answer text and emotional state to obtain the answer speech; and conduct interactive question and answer based on the answer speech. The text generated by multimodal understanding plus the predicted emotion and emotion level will be sent to the speech synthesis model for synthesis to synthesize the corresponding answer. The whole process is streaming. The speech model and multimodal understanding model work simultaneously.

[0085] S4, when the current question instruction does not contain a pronoun, or when the referent of the pronoun in the current question instruction is known based on historical questions and answers, interactive question and answer is performed based on the large voice model, the current question instruction and the current image.

[0086] The interactive question and answer device provided by the present invention is described below. The interactive question and answer device described below and the interactive question and answer method described above can be referenced to each other.

[0087] Based on any of the above embodiments, Figure 8 is a schematic diagram of the structure of the interactive question-answering device provided by the present invention. Figure 8 As shown, the interactive question-answering device comprises: An acquisition unit 810 is used to acquire a question instruction and an image containing a directional object; A generating unit 820 is used to associate the pronoun in the question instruction with the object indicated by the directional object, obtain the object referred to by the pronoun, and generate a response text based on the object referred to; The interactive unit 830 is used to conduct interactive question and answer based on the answer text.

[0088] The device provided by the embodiment of the present invention can finely identify and understand the user's referential questions by associating the pronouns in the question instructions with the specific objects indicated by the directional objects, thereby correctly understanding the user's intentions, and ensuring the alignment of multimodal information by intelligently parsing the pronouns, so that the answer content is closer to the natural logic of human dialogue, thereby improving the fluency of interaction and user experience.

[0089] Based on any of the above embodiments, the generating unit is specifically used for: Performing object detection on the image to obtain the directional object and the position of each object; Based on the directional object and the positions of each object, the pronoun in the question instruction is associated with the object indicated by the directional object to obtain the object referred to by the pronoun.

[0090] Based on any of the above embodiments, the generating unit is specifically used for: Based on the positions of the directional object and each object, determining the distance between the directional object and each object, and the indication direction of the directional object; Based on the distance and the indicated direction, the pronoun in the question instruction is associated with the object indicated by the directional object to obtain the object referred to by the pronoun.

[0091] Based on any of the above embodiments, the interaction unit is specifically used for: Based on the question instruction, predicting the emotional state contained in the answer text; Perform speech synthesis based on the answer text and the emotional state to obtain an answer speech; Based on the answer voice, interactive question and answer is performed.

[0092] Based on any of the above embodiments, it also includes a voice interaction unit, which is used to: When the current question instruction does not contain a pronoun, or the referent of the pronoun in the current question instruction is known based on historical questions and answers, interactive questions and answers are performed based on the large voice model, the current question instruction and the current image.

[0093] Based on any of the above embodiments, the generating unit is specifically used for: Based on a large multimodal understanding model, the pronoun in the question instruction is associated with the object indicated by the directional object to obtain the referent of the pronoun, and a response text is generated based on the referent. The large multimodal understanding model is obtained by training a large language model in stages based on sample question instructions and sample images.

[0094] Based on any of the above embodiments, the question instruction includes a text instruction or a voice instruction, and the image includes a single image, multiple images containing dynamic directional objects in a single scene, multiple images containing dynamic directional objects across scenes, and multiple video frames with semantic associations.

[0095] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the interactive question-answering method, which includes: obtaining a question instruction and an image containing a directional object; associating the reference word in the question instruction with the object indicated by the directional object to obtain the reference object of the reference word, and generating an answer text based on the reference object; and performing interactive question-answering based on the answer text.

[0096] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0097] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the interactive question and answer method provided by the above-mentioned methods, which includes: obtaining a question instruction and an image containing a directional object; associating the pronoun in the question instruction with the object indicated by the directional object to obtain the reference object of the pronoun, and generating an answer text based on the reference object; and conducting interactive question and answer based on the answer text.

[0098] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the interactive question-and-answer method provided by the above-mentioned methods, the method comprising: obtaining a question instruction and an image containing a directional object; associating the pronoun in the question instruction with the object indicated by the directional object to obtain the reference object of the pronoun, and generating an answer text based on the reference object; and conducting interactive question-and-answer based on the answer text.

[0099] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An interactive question-answering method, characterized in that: include: Obtain questioning instructions and images containing pointed objects; Associating the pronoun in the question instruction with the object indicated by the directional object to obtain the referent of the pronoun, and generating an answer text based on the referent; Based on the answer text, interactive question and answer is performed.

2. The interactive question-answering method according to claim 1, characterized in that: The step of associating the pronoun in the question instruction with the object indicated by the directional object to obtain the object referred to by the pronoun includes: Performing object detection on the image to obtain the directional object and the position of each object; Based on the directional object and the positions of each object, the pronoun in the question instruction is associated with the object indicated by the directional object to obtain the object referred to by the pronoun.

3. The interactive question-answering method according to claim 2, characterized in that: The step of associating the pronoun in the question instruction with the object indicated by the directional object based on the directional object and the position of each object to obtain the object referred to by the pronoun includes: Based on the positions of the directional object and each object, determining the distance between the directional object and each object, and the indication direction of the directional object; Based on the distance and the indicated direction, the pronoun in the question instruction is associated with the object indicated by the directional object to obtain the object referred to by the pronoun.

4. The interactive question-answering method according to any one of claims 1 to 3, characterized in that: The interactive question-answering based on the answer text includes: Based on the question instruction, predicting the emotional state contained in the answer text; Perform speech synthesis based on the answer text and the emotional state to obtain an answer speech; Based on the answer voice, interactive question and answer is performed.

5. The interactive question-answering method according to any one of claims 1 to 3, characterized in that: The method further comprises: When the current question instruction does not contain a pronoun, or the referent of the pronoun in the current question instruction is known based on historical questions and answers, interactive questions and answers are performed based on the large voice model, the current question instruction and the current image.

6. The interactive question-answering method according to any one of claims 1 to 3, characterized in that: The step of associating the pronoun in the question instruction with the object indicated by the directional object to obtain the object referred to by the pronoun, and generating an answer text based on the object referred to includes: Based on a large multimodal understanding model, the pronoun in the question instruction is associated with the object indicated by the directional object to obtain the referent of the pronoun, and a response text is generated based on the referent. The large multimodal understanding model is obtained by training a large language model in stages based on sample question instructions and sample images.

7. The interactive question-answering method according to any one of claims 1 to 3, characterized in that: The question instruction includes a text instruction or a voice instruction, and the image includes a single image, multiple images containing dynamic directional objects in a single scene, multiple images containing dynamic directional objects across scenes, and multiple video frames with semantic associations.

8. An interactive question-answering device, characterized in that: include: An acquisition unit, used for acquiring a question instruction and an image containing a directional object; A generating unit, configured to associate the pronoun in the question instruction with the object indicated by the directional object, obtain the object referred to by the pronoun, and generate an answer text based on the object referred to; The interactive unit is used for conducting interactive question and answer based on the answer text.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the interactive question-and-answer method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the interactive question-answering method according to any one of claims 1 to 7 is implemented.

Images