Question and answer processing method and device, equipment and storage medium

Abstract

The invention provides a question and answer processing method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: when an input target question is answered, matching a question text corresponding to the target question with text data corresponding to various modal data in a question and answer database, and determining target text data matched with the question text from the various text data, the multiple modal data comprises at least two of text state data, video state data, audio state data, picture state data and picture table state data; based on the question text and the target text data, determining an answer text corresponding to the question text; the question text and the answer text are input into the dichotomy model to obtain the dichotomy result, the dichotomy result is used for indicating whether to output the modal data corresponding to the answer text, and the target question is answered according to the dichotomy result, so that the question input by the user can be flexibly answered, and the answering accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a question-answering processing method, apparatus, device and storage medium. Background Art

[0002] In the insurance field, the insurance data stored in the question-and-answer database includes not only text data, but also other modal data, such as audio data, image data, chart data, or video data.

[0003] At present, for answering scenarios in the insurance field, after receiving questions input by users, Multimodal Embedding technology is usually used to match the questions with multiple modal data in the question and answer data respectively to check the answer data for answering. However, the accuracy of the answer data retrieved in this way is low and it is not possible to answer the questions input by users well.

[0004] However, considering that in actual application, in some definitional Q&A scenarios or Q&A scenarios such as insurance agreements, it is usually necessary to answer only in the form of text answers; while in other operational Q&A scenarios or registration and activation of insurance policies, it is usually necessary to answer in combination with image data or video data. Therefore, how to flexibly answer questions input by users and improve the accuracy of answers is a technical problem that technicians in this field urgently need to solve. Summary of the invention

[0005] The present application provides a question-answering processing method, apparatus, device and storage medium to solve the defects in the prior art in how to flexibly answer questions input by users, thereby improving the accuracy of answers.

[0006] This application provides a question-answering processing method, including: Receive input target question; Matching the question text corresponding to the target question with the text data corresponding to each of the multiple modal data in the question-answering database, and determining the target text data matching the question text from the multiple text data, wherein the multiple modal data include at least two of text data, video data, audio data, picture data, and chart data, and the target text data includes text data corresponding to at least one modal data; Based on the question text and the target text data, determining an answer text corresponding to the question text; The question text and the answer text are input into a binary classification model to obtain a binary classification result, wherein the binary classification result is used to indicate whether to output the modal data corresponding to the answer text, and the target question is answered according to the binary classification result.

[0007] According to a question-answer processing method provided by the present application, determining the answer text corresponding to the question text based on the question text and the target text data includes: Splicing the question text and the target text data to obtain a spliced ​​text; The concatenated text is input into a pre-built large model to obtain the answer text corresponding to the question text; wherein the large model is trained based on a concatenated text sample obtained by connecting a question text sample and a target text data sample, and an answer text sample corresponding to the question text sample.

[0008] According to a question-answer processing method provided by the present application, the question text corresponding to the target question is matched with the text data corresponding to each of the multiple modal data in the question-answer database, and the target text data matching the question text is determined from the multiple text data, including: Input the question text and the multiple text data into the BGE-Embeddings model to obtain a question text vector corresponding to the question text and a data text vector corresponding to each of the multiple text data; The similarities between the question text vector and the data text vectors corresponding to the multiple text data are respectively determined, and the target data text is determined from the multiple text data based on the similarities.

[0009] According to a question-answering processing method provided by the present application, answering the target question according to the binary classification result includes: In the case where the binary classification result indicates that the modal data corresponding to the answer text is not to be output, responding with the answer text as response information; In the case where the binary classification result indicates that the modal data corresponding to the answer text is to be output, both the answer text and the modal data corresponding to the answer text are responded as response information.

[0010] According to a question-answer processing method provided by the present application, the method further includes: In the case where the multiple modal data include picture data, the picture data is processed using GPT-4V technology to generate picture summary text and key content text corresponding to the picture data; The picture summary text and the key content text are determined as text data corresponding to the picture state data.

[0011] According to a question-answer processing method provided by the present application, the method further includes: In the case where the multiple modal data include video state data, the video state data is processed using GPT-4V technology to generate a video summary text and a subtitle text corresponding to the video state data; The video summary text and the subtitle text are determined as text data corresponding to the video state data.

[0012] The present application also provides a question-answer processing device, comprising: A receiving unit, used for receiving an input target question; a matching unit, for matching the question text corresponding to the target question with the text data corresponding to each of the multiple modal data in the question-answering database, and determining the target text data matching the question text from the multiple text data, wherein the multiple modal data include at least two of text data, video data, audio data, picture data and chart data, and the target text data include text data corresponding to at least one modal data; A first processing unit, configured to determine an answer text corresponding to the question text based on the question text and the target text data; The second processing unit is used to input the question text and the answer text into a binary classification model to obtain a binary classification result, wherein the binary classification result is used to indicate whether to output the modal data corresponding to the answer text, and to answer the target question according to the binary classification result.

[0013] According to a question-answer processing device provided by the present application, the first processing unit is used to determine the answer text corresponding to the question text based on the question text and the target text data, including: Splicing the question text and the target text data to obtain a spliced ​​text; The concatenated text is input into a pre-built large model to obtain the answer text corresponding to the question text; wherein the large model is trained based on a concatenated text sample obtained by connecting a question text sample and a target text data sample, and an answer text sample corresponding to the question text sample.

[0014] According to a question-and-answer processing device provided by the present application, the matching unit is used to match the question text corresponding to the target question with the text data corresponding to each of the multiple modal data in the question-and-answer database, and determining the target text data matching the question text from the multiple text data includes: Input the question text and the multiple text data into the BGE-Embeddings model to obtain a question text vector corresponding to the question text and a data text vector corresponding to each of the multiple text data; The similarities between the question text vector and the data text vectors corresponding to the multiple text data are respectively determined, and the target data text is determined from the multiple text data based on the similarities.

[0015] According to a question-answer processing device provided by the present application, the second processing unit is used to answer the target question according to the binary classification result, including: In the case where the binary classification result indicates that the modal data corresponding to the answer text is not to be output, responding with the answer text as response information; In the case where the binary classification result indicates that the modal data corresponding to the answer text is to be output, both the answer text and the modal data corresponding to the answer text are responded as response information.

[0016] According to a question-answer processing device provided by the present application, the device also includes: A first generating unit is configured to, when the multiple modal data include picture data, process the picture data using the GPT-4V technology to generate a picture summary text and a key content text corresponding to the picture data; The first determining unit is used to determine the picture summary text and the key content text as text data corresponding to the picture state data.

[0017] According to a question-answer processing device provided by the present application, the device also includes: A second generating unit is configured to, when the multiple modal data include video state data, process the video state data using the GPT-4V technology to generate a video summary text and a subtitle text corresponding to the video state data; The second determining unit is used to determine the video summary text and the subtitle text as text data corresponding to the video state data.

[0018] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described question and answer processing methods when executing the computer program.

[0019] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the question-and-answer processing methods described above.

[0020] The present application also provides a computer program product, including a computer program, which implements any of the question-and-answer processing methods described above when executed by a processor.

[0021] The present application provides a question-answering processing method, device, equipment and storage medium. When answering an input target question, the question text corresponding to the target question is first matched with the text data corresponding to each of the multiple modal data in the question-answering database, and the target text data matching the question text is determined from the multiple text data. The multiple modal data include at least two of text state data, video state data, audio state data, picture state data and chart state data. The target text data includes text data corresponding to at least one modal data; and based on the question text and the target text data, the answer text corresponding to the question text is determined; and the question text and the answer text are input into a binary classification model to obtain a binary classification result, which is used to indicate whether to output the modal data corresponding to the answer text, and the target question is answered according to the binary classification result. In this way, after determining the answer text corresponding to the question text, the answer text is not directly used as the answer information for answering, but the binary classification model is combined to jointly determine how to answer the target question, so that the question input by the user can be flexibly answered, thereby improving the accuracy of the answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A flowchart of a question-and-answer processing method provided in an embodiment of the present application.

[0024] Figure 2 A schematic diagram of a question-and-answer framework provided in an embodiment of the present application.

[0025] Figure 3 A structural diagram of a question and answer processing device provided in an embodiment of the present application.

[0026] Figure 4 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] In the embodiments of the present application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the text description of the present application, the character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0029] At present, for answering scenarios in the insurance field, after receiving questions input by users, Multimodal Embedding technology is usually used to match the questions with multiple modal data in the question and answer data respectively to check the answer data for answering. However, the accuracy of the answer data retrieved in this way is low and it is not possible to answer the questions input by users well.

[0030] However, considering that in actual application, in some definitional question and answer scenarios or question and answer scenarios such as insurance agreements, it is usually only necessary to respond in the form of text answers; while in other operational question and answer scenarios or registration and policy activation scenarios, it is usually necessary to combine image data or video data to respond.

[0031] Therefore, in order to flexibly answer questions input by users and thus improve the accuracy of answers, the embodiment of the present application provides a question-answering processing method that can be applied to question-answering scenarios. The execution subject of the method can be a computer, a server, or an electronic device such as a mobile robot or a mobile terminal, or a question-answering processing device disposed in the electronic device, which can be implemented by software, hardware, or a combination of the two.

[0032] The question-answer processing method provided by the present application will be described in detail below through the following specific embodiments. It is understandable that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0033] Figure 1 A flowchart of a question-answering processing method provided in an embodiment of the present application is provided. For example, see Figure 1As shown, the question-answering processing method may include: S101, receiving an input target question.

[0034] For example, the target question can be input in text form or in voice or other forms, and can be set according to actual needs.

[0035] For example, the target question may be “What are the terms and conditions of the insurance agreement?” or “How do I fill in the policyholder information?”, etc. The target question may be set according to actual needs.

[0036] S102. Match the question text corresponding to the target question with the text data corresponding to the various modal data in the question and answer database respectively, and determine the target text data matching the question text from the various text data. The various modal data include at least two of text data, video data, audio data, picture data and chart data, and the target text data includes text data corresponding to at least one modal data.

[0037] Among them, the question and answer database stores the text data corresponding to each of the multiple modal data, that is, the multiple modal data are converted into corresponding text data in advance, so that when answering, the question text corresponding to the target question can be directly matched with the text data corresponding to each of the multiple modal data.

[0038] Taking multiple modal data including text data, video data, audio data, image data and chart data as an example, it can be understood that there is no need to convert the text data; when converting the video data into the corresponding text data, in one possible implementation method, the GPT-4V technology can be used to process the video data to generate a video summary text and subtitle text corresponding to the video data; and the video summary text and subtitle text are determined as the text data corresponding to the video data; or, in another possible implementation method, a picture including text can be captured from the video through video editing software or a screenshot tool, and an optical character recognition (OCR) tool can be used to perform OCR recognition on the screenshot picture to convert the picture into text data; of course, other methods can also be used to obtain the text data corresponding to the video data, which can be set according to actual needs.

[0039] In the case where multiple modal data include audio data, when converting the audio data into corresponding text data, in one possible implementation, speech recognition (SR) technology or automatic speech recognition (ASR) can be used to convert the audio data into corresponding text data; in another possible implementation, the SpeechRecognition library can be used to convert the audio data into corresponding text data; of course, other methods can also be used to obtain text data corresponding to video data, which can be set according to actual needs.

[0040] Among them, an example using the SpeechRecognition library can be shown as follows: Python

[0041] import speech_recognition as sr

[0042] r = sr.Recognizer()

[0043] with sr.Microphone() as source: print("Say something!")

[0044] audio = r.listen(source)

[0045] text = r.recognize_google(audio, language='en-US')

[0046] print("You said: {}".format(text))

[0047] In the case where multiple modal data include image data, when converting the image data into corresponding text data, in one possible implementation, the GPT-4V technology can be used to process the image data to generate image summary text and key content text corresponding to the image data; and the image summary text and key content text are determined as text data corresponding to the image data; or, in another possible implementation, the image data can be subjected to OCR recognition through an OCR tool to convert the image data into text data; of course, other methods can also be used to obtain text data corresponding to the image data, which can be set according to actual needs.

[0048] In the case where multiple modal data include chart-like data, when converting the chart-like data into corresponding text data, in one possible implementation, if the chart-like data is spreadsheet data such as Excel, Google Sheets, etc., the data table in the chart-like data can be directly copied and pasted into a text editor, such as Notepad, Word, etc., to obtain the text data corresponding to the chart-like data; in another possible implementation, if the chart-like data exists in the form of a picture, the picture-like data can be OCR recognized by an OCR tool to convert the picture-like data into text data; of course, other methods can also be used to obtain the text data corresponding to the chart-like data, which can be set according to actual needs.

[0049] In combination with the above description, after respectively obtaining the text data corresponding to each of the multiple modal data, the text data corresponding to each of the multiple modal data can be stored in a question and answer database, so that after receiving the input target question, the question text can be matched with the text data corresponding to each of the multiple modal data in the question and answer database, and the target text data matching the question text can be determined from the multiple text data.

[0050] Among them, the target text data may include text data corresponding to at least one type of modal data, for example, text data corresponding to video data and text data corresponding to image data; or, text data corresponding to image data and text data corresponding to chart data, etc., which can be set according to actual needs.

[0051] S103: Determine the answer text corresponding to the question text based on the question text and the target text data.

[0052] After determining the answer text corresponding to the question text based on the question text and the target text data, in the embodiment of the present application, the answer text is not directly used as the response information for response, but it is fully considered that in the actual application process, in some definitional question and answer scenarios or question and answer scenarios such as insurance agreements, it is usually only necessary to respond in the form of text answers; while in other operation-type question and answer scenarios or registration-type activation policy scenarios, it is usually necessary to respond in combination with image data or video data. Therefore, it is possible to combine the binary classification model to jointly determine how to respond to the target question, that is, to execute the following S104, so that the questions input by the user can be flexibly responded to, thereby improving the accuracy of the response.

[0053] S104. Input the question text and the answer text into a binary classification model to obtain a binary classification result. The binary classification result is used to indicate whether to output the modal data corresponding to the answer text, and answer the target question according to the binary classification result.

[0054] For example, in the embodiment of the present application, when answering the target question according to the binary classification result, two possible scenarios may be included: In one possible scenario, if the binary classification result indicates that the modal data corresponding to the answer text is not to be output, the answer text can be directly used as the response information.

[0055] In another possible scenario, when the binary classification result indicates the output of the modal data corresponding to the answer text, both the answer text and the modal data corresponding to the answer text can be used as response information.

[0056] For example, if the answer text includes text data corresponding to some video data and text data corresponding to audio data, the corresponding video data and audio data can be output together with the answer text to facilitate better answering.

[0057] It can be seen that in the embodiment of the present application, when answering the input target question, the question text corresponding to the target question is first matched with the text data corresponding to each of the multiple modal data in the question-answering database, and the target text data matching the question text is determined from the multiple text data, and the multiple modal data include at least two of text state data, video state data, audio state data, picture state data and chart state data, and the target text data includes text data corresponding to at least one modal data; and based on the question text and the target text data, the answer text corresponding to the question text is determined; and the question text and the answer text are input into the binary classification model to obtain a binary classification result, which is used to indicate whether to output the modal data corresponding to the answer text, and the target question is answered according to the binary classification result. In this way, after determining the answer text corresponding to the question text, the answer text is not directly used as the answer information for answering, but combined with the binary classification model to jointly determine how to answer the target question, and the user input question can be flexibly answered, thereby improving the accuracy of the answer.

[0058] Based on the above Figure 1 In the embodiment shown, in the above S102, the question text corresponding to the target question is matched with the text data corresponding to each of the multiple modal data in the question and answer database, and the target text data matching the question text is determined from the multiple text data, which may include: The question text and multiple text data are input into the BGE-Embeddings model to obtain the question text vector corresponding to the question text and the data text vectors corresponding to the multiple text data; the similarities between the question text vector and the data text vectors corresponding to the multiple text data are determined respectively, and the target data text is determined from the multiple text data based on the similarities.

[0059] Among them, the BGE-Embeddings model is a technology that converts text data into low-dimensional dense vectors. These vectors can capture the semantic information of the text and facilitate computer understanding and processing.

[0060] For example, when training the BGE-Embeddings model, contrastive learning can be used for training, specifically including: determining training samples. For example, the training samples may include problem samples, positive samples corresponding to the problem samples, and multiple negative samples. Among them, the problem samples may be problems in the insurance industry queried by users, the positive samples are text content related to the questions, and the negative samples are irrelevant text content. When training based on training samples, InfoNCE loss can be used. The InfoNCE loss is a contrastive loss function commonly used in self-supervised learning tasks, especially in the framework of contrastive learning. It aims to narrow the distance between positive samples (similar samples) and push the distance between negative samples (dissimilar samples) to learn the effective representation of data. The specific formula can be shown in the following formula:

[0061] in, represents the InfoNCE loss, Represents a sample of problems, represents a positive sample, represents negative samples, represents the number of negative samples, represents the first of the K negative samples Negative samples.

[0062] In the BGE-Embeddings model, after obtaining the question text vector and the data text vector corresponding to the question text, the similarity between the question text vector and the data text vector can be determined, which can be recorded as sim(x, y), usually calculated using dot product or cosine similarity. Based on the similarity, the text data with a similarity greater than a similarity threshold is determined as the target data text. In this way, the target text data matching the question text can be determined from the text data corresponding to multiple modal data.

[0063] Based on any of the above embodiments, in the above S103, when determining the answer text corresponding to the question text based on the question text and the target text data, the question text and the target text data can be spliced ​​to obtain the spliced ​​text; and the spliced ​​text can be input into the pre-built large model to obtain the answer text corresponding to the question text, so that after obtaining the answer text, the question text and the answer text can be input into the binary classification model to obtain the binary classification result, and the target question can be answered according to the binary classification result. In this way, after determining the answer text corresponding to the question text, the answer text is not directly used as the answer information for answering, but combined with the binary classification model to jointly determine how to answer the target question, and the questions input by the user can be flexibly answered, thereby improving the accuracy of the answer.

[0064] Among them, the large model is trained based on the concatenated text samples obtained by connecting the question text samples and the target text data samples, and the answer text samples corresponding to the question text samples.

[0065] In combination with the above description, in the embodiment of the present application, when answering questions, please refer to Figure 2 As shown, Figure 2 A schematic diagram of a framework for question-and-answer answering provided in an embodiment of the present application first receives a target question input by a user, inputs the question text corresponding to the target question and the text data corresponding to each of the multimodal data stored in the question-and-answer database into the BGE-Embeddings model, obtains a question text vector corresponding to the question text and a data text vector corresponding to each of the multiple text data; and based on the similarity between the question text vector and the data text vector corresponding to each of the multiple text data, determines the target data text from the multiple text data. By way of example, the target data text may include text data corresponding to at least one modality data, for example, including text data corresponding to part of the video state data and text data corresponding to the audio state data.

[0066] The question text and target text data are concatenated to obtain a concatenated text; the concatenated text is input into a pre-built large model to obtain an answer text corresponding to the question text; the question text and the answer text are input into a binary classification model to obtain a binary classification result, and the target question is answered based on the binary classification result.

[0067] If the binary classification result indicates that the modal data corresponding to the answer text is not output, the answer text can be directly used as the response information for response; on the contrary, if the binary classification result indicates that the modal data corresponding to the answer text is output, the answer text and the modal data corresponding to the answer text can be used as the response information for response. For example, when outputting the answer text, the corresponding video state data and audio state data can be output at the same time, so that the questions input by the user can be flexibly responded to, thereby improving the accuracy of the response.

[0068] The question and answer processing device provided in the present application is described below. The question and answer processing device described below and the question and answer processing method described above can be referenced to each other.

[0069] Figure 3 This is a structural diagram of a question-answering processing device provided in an embodiment of the present application. For example, see Figure 3 As shown, the question-answer processing device 30 may include: A receiving unit 301 is used to receive an input target question; A matching unit 302 is used to match the question text corresponding to the target question with the text data corresponding to each of the multiple modal data in the question-answer database, and determine the target text data matching the question text from the multiple text data, wherein the multiple modal data include at least two of text data, video data, audio data, picture data and chart data, and the target text data includes text data corresponding to at least one modal data; A first processing unit 303 is used to determine an answer text corresponding to the question text based on the question text and the target text data; The second processing unit 304 is used to input the question text and the answer text into a binary classification model to obtain a binary classification result, wherein the binary classification result is used to indicate whether to output the modal data corresponding to the answer text, and to answer the target question according to the binary classification result.

[0070] For example, in the embodiment of the present application, the first processing unit 303 is used to determine the answer text corresponding to the question text based on the question text and the target text data, including: Splicing the question text and the target text data to obtain a spliced ​​text; The concatenated text is input into a pre-built large model to obtain the answer text corresponding to the question text; wherein the large model is trained based on a concatenated text sample obtained by connecting a question text sample and a target text data sample, and an answer text sample corresponding to the question text sample.

[0071] For example, in an embodiment of the present application, the matching unit 302 is used to match the question text corresponding to the target question with the text data corresponding to each of the multiple modal data in the question and answer database, and determining the target text data matching the question text from the multiple text data includes: Input the question text and the multiple text data into the BGE-Embeddings model to obtain a question text vector corresponding to the question text and a data text vector corresponding to each of the multiple text data; The similarities between the question text vector and the data text vectors corresponding to the multiple text data are respectively determined, and the target data text is determined from the multiple text data based on the similarities.

[0072] For example, in the embodiment of the present application, the second processing unit 304 is used to answer the target question according to the binary classification result, including: In the case where the binary classification result indicates that the modal data corresponding to the answer text is not to be output, responding with the answer text as response information; In the case where the binary classification result indicates that the modal data corresponding to the answer text is to be output, both the answer text and the modal data corresponding to the answer text are responded as response information.

[0073] For example, in the embodiment of the present application, the question and answer processing device 30 further includes: A first generating unit is configured to, when the multiple modal data include picture data, process the picture data using the GPT-4V technology to generate a picture summary text and a key content text corresponding to the picture data; The first determining unit is used to determine the picture summary text and the key content text as text data corresponding to the picture state data.

[0074] For example, in the embodiment of the present application, the question and answer processing device 30 further includes: A second generating unit is configured to, when the multiple modal data include video state data, process the video state data using the GPT-4V technology to generate a video summary text and a subtitle text corresponding to the video state data; The second determining unit is used to determine the video summary text and the subtitle text as text data corresponding to the video state data.

[0075] The question and answer processing device 30 provided in the embodiment of the present application can execute the technical solution of the question and answer processing method in any of the above-mentioned embodiments. Its implementation principle and beneficial effects are similar to the implementation principle and beneficial effects of the question and answer processing method. Please refer to the implementation principle and beneficial effects of the question and answer processing method, and no further details will be given here.

[0076] Figure 4 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application, such as Figure 4As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other through the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute the question and answer processing method, which includes: receiving an input target question; matching the question text corresponding to the target question with the text data corresponding to each of the multiple modal data in the question and answer database, and determining the target text data matching the question text from the multiple text data, wherein the multiple modal data include at least two of text data, video data, audio data, picture data and chart data, and the target text data includes text data corresponding to at least one modal data; based on the question text and the target text data, determining the answer text corresponding to the question text; inputting the question text and the answer text into a binary classification model to obtain a binary classification result, wherein the binary classification result is used to indicate whether to output the modal data corresponding to the answer text, and answering the target question according to the binary classification result.

[0077] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0078] On the other hand, the present application also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the question and answer processing method provided by the above-mentioned methods, which includes: receiving an input target question; matching the question text corresponding to the target question with the text data corresponding to each of the multiple modal data in the question and answer database, and determining the target text data matching the question text from the multiple text data, the multiple modal data including at least two of text data, video data, audio data, picture data and chart data, and the target text data including text data corresponding to at least one modal data; based on the question text and the target text data, determining the answer text corresponding to the question text; inputting the question text and the answer text into a binary classification model to obtain a binary classification result, the binary classification result is used to indicate whether to output the modal data corresponding to the answer text, and answering the target question according to the binary classification result.

[0079] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the question and answer processing method provided by the above-mentioned methods, the method comprising: receiving an input target question; matching the question text corresponding to the target question with the text data corresponding to each of the multiple modal data in the question and answer database, and determining the target text data matching the question text from the multiple text data, the multiple modal data comprising at least two of text data, video data, audio data, picture data and chart data, and the target text data comprising text data corresponding to at least one modal data; determining the answer text corresponding to the question text based on the question text and the target text data; inputting the question text and the answer text into a binary classification model to obtain a binary classification result, the binary classification result being used to indicate whether to output the modal data corresponding to the answer text, and answering the target question according to the binary classification result.

[0080] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0081] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A question-answering processing method, characterized in that: include: Receive input target question; Matching the question text corresponding to the target question with the text data corresponding to each of the multiple modal data in the question-answering database, and determining the target text data matching the question text from the multiple text data, wherein the multiple modal data include at least two of text data, video data, audio data, picture data, and chart data, and the target text data includes text data corresponding to at least one modal data; Based on the question text and the target text data, determining an answer text corresponding to the question text; The question text and the answer text are input into a binary classification model to obtain a binary classification result, wherein the binary classification result is used to indicate whether to output the modal data corresponding to the answer text, and the target question is answered according to the binary classification result.

2. The question-answer processing method according to claim 1, characterized in that: The step of determining the answer text corresponding to the question text based on the question text and the target text data includes: Splicing the question text and the target text data to obtain a spliced ​​text; The concatenated text is input into a pre-built large model to obtain the answer text corresponding to the question text; wherein the large model is trained based on a concatenated text sample obtained by connecting a question text sample and a target text data sample, and an answer text sample corresponding to the question text sample.

3. The question-answer processing method according to claim 1 or 2, characterized in that: The step of matching the question text corresponding to the target question with the text data corresponding to the various modal data in the question-answer database, and determining the target text data matching the question text from the various text data comprises: Input the question text and the multiple text data into the BGE-Embeddings model to obtain a question text vector corresponding to the question text and a data text vector corresponding to each of the multiple text data; The similarities between the question text vector and the data text vectors corresponding to the multiple text data are respectively determined, and the target data text is determined from the multiple text data based on the similarities.

4. The question-answer processing method according to claim 1 or 2, characterized in that: The answering the target question according to the binary classification result includes: In the case where the binary classification result indicates that the modal data corresponding to the answer text is not to be output, responding with the answer text as response information; In the case where the binary classification result indicates that the modal data corresponding to the answer text is to be output, both the answer text and the modal data corresponding to the answer text are responded as response information.

5. The question-answer processing method according to claim 1 or 2, characterized in that: The method further comprises: In the case where the multiple modal data include picture data, the picture data is processed using GPT-4V technology to generate picture summary text and key content text corresponding to the picture data; The picture summary text and the key content text are determined as text data corresponding to the picture state data.

6. The question-answer processing method according to claim 1 or 2, characterized in that: The method further comprises: In the case where the multiple modal data include video state data, the video state data is processed using GPT-4V technology to generate a video summary text and a subtitle text corresponding to the video state data; The video summary text and the subtitle text are determined as text data corresponding to the video state data.

7. A question-answer processing device, characterized in that: include: A receiving unit, used for receiving an input target question; a matching unit, for matching the question text corresponding to the target question with the text data corresponding to each of the multiple modal data in the question-answering database, and determining the target text data matching the question text from the multiple text data, wherein the multiple modal data include at least two of text data, video data, audio data, picture data and chart data, and the target text data include text data corresponding to at least one modal data; A first processing unit, configured to determine an answer text corresponding to the question text based on the question text and the target text data; The second processing unit is used to input the question text and the answer text into a binary classification model to obtain a binary classification result, wherein the binary classification result is used to indicate whether to output the modal data corresponding to the answer text, and to answer the target question according to the binary classification result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the question and answer processing method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the question and answer processing method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the question and answer processing method according to any one of claims 1 to 6 is implemented.

Images