Voice answering method, device, electronic device and storage medium

Abstract

The present invention provides a method, device, electronic device and storage medium for answering questions by voice, wherein the method comprises: obtaining a user's answering voice for a current question; based on the characteristics of the alternative answers to the current question in terms of answer numbers and / or answer contents, converting the answer to a voice recognition text of the answering voice to obtain the user's voice answer; and determining the answer result based on the voice answer and the standard answer to the current question. The method, device, electronic device and storage medium for answering questions by voice provided by the present invention are not restricted by specific trigger words, making the user's answering more flexible, and can effectively improve the user experience while ensuring the accuracy of voice answer recognition.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a voice answering method, device, electronic equipment and storage medium. Background Art

[0002] Tool-type educational products provide a voice answering function. Users can complete online answers simply by listening to the questions and then giving voice responses, which can effectively improve the convenience of answering questions for users.

[0003] The current voice answering method is to trigger the answer according to specific vocabulary. For example, you must voice answer A / B / C / D before you can answer the question by voice. If it fails to be triggered many times, it will easily make users bored, resulting in a poor user experience. Summary of the invention

[0004] The present invention provides a voice answering method, device, electronic device and storage medium, which are used to solve the defect in the prior art that answering is triggered according to specific vocabulary, resulting in a poor user experience.

[0005] The present invention provides a voice answering method, comprising:

[0006] Get the user's answer to the current question;

[0007] Based on the characteristics of the candidate answers of the current question in terms of answer numbers and / or answer contents, converting the speech recognition text of the answering speech into an answer to obtain the user's voice answer;

[0008] The answer result is determined based on the voice answer and the standard answer to the current question.

[0009] According to a voice answering method provided by the present invention, the answer conversion is performed on the voice recognition text of the answering voice based on the characteristics of the alternative answers of the current question in terms of answer number and / or answer content to obtain the voice answer of the user, including:

[0010] Convert the answer number text in the speech recognition text into a number format to obtain a speech answer number that conforms to the format of the answer number of the alternative answer;

[0011] and / or, extracting entities of a preset type from the speech recognition text, and using the extracted entities as speech answer content, wherein the preset type is determined based on the characteristics of the alternative answers in terms of the answer content;

[0012] The voice answer is determined based on the voice answer number and / or the voice answer content.

[0013] According to a voice answering method provided by the present invention, the preset type is determined based on the entity type of the answer content in the alternative answers.

[0014] According to a voice answering method provided by the present invention, the answering result is determined based on the voice answer and the standard answer to the current question, including:

[0015] If there is voice answer content in the voice answer, convert the voice answer content into pinyin to be reviewed;

[0016] The answer result is determined based on the pinyin to be reviewed and the pinyin of the standard answer.

[0017] According to a voice answering method provided by the present invention, the answering result is determined based on the voice answer and the standard answer to the current question, including:

[0018] Based on the finger answer and / or the sight answer, the voice answer is modified to obtain the user answer, wherein the finger answer is determined based on the finger image of the user pointing to the answer screen, and the sight answer is determined based on the face image of the user;

[0019] The answer result is determined based on the user's answer and the standard answer to the current question.

[0020] According to a voice answering method provided by the present invention, the finger answer is determined based on the following steps:

[0021] Obtaining a finger pointing vector of the finger image;

[0022] The finger answer is determined based on the placement angle of the answer screen, the area occupied by each alternative answer row on the answer screen, and the projection position of the finger pointing amount on the answer screen.

[0023] According to a voice answering method provided by the present invention, the sight answer is determined based on the following steps:

[0024] Determining the sight direction of the human eye based on the human eye features of the facial image;

[0025] Determining the spatial position of the human eye based on the key point information of the facial image;

[0026] The gaze answer is determined based on the gaze direction and spatial position of the human eye.

[0027] The present invention also provides a voice answering device, comprising:

[0028] A voice acquisition unit, used to acquire the user's answer voice for the current question;

[0029] An answer conversion unit, configured to convert the voice recognition text of the answering voice into an answer based on the characteristics of the candidate answers of the current question in terms of answer numbers and / or answer contents, so as to obtain the voice answer of the user;

[0030] The result determination unit is used to determine the answer result based on the voice answer and the standard answer to the current question.

[0031] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the voice answering method as described above is implemented.

[0032] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the voice answering method as described in any one of the above is implemented.

[0033] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned voice answering methods.

[0034] The voice answering method, device, electronic device and storage medium provided by the present invention convert the voice recognition text of the answering voice into an answer based on the characteristics of the alternative answers of the current question in terms of answer number and / or answer content, and obtain the user's voice answer. The answering method is not limited by specific trigger words, making the user's answering more flexible, and can effectively improve the user experience while ensuring the accuracy of voice answer recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is a flow chart of the voice answering method provided by the present invention;

[0037] Figure 2 is a flow chart of step 120 in the voice answering method provided by the present invention;

[0038] Figure 3 This is one of the flowchart diagrams of step 130 in the voice answering method provided by the present invention;

[0039] Figure 4This is the second flow chart of step 130 in the voice answering method provided by the present invention;

[0040] Figure 5 It is a flow chart of the finger answer determination method provided by the present invention;

[0041] Figure 6 It is a flow chart of the sight line answer determination method provided by the present invention;

[0042] Figure 7 It is a structural schematic diagram of the voice answering device provided by the present invention;

[0043] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The existing voice answering method is usually triggered by answering questions according to specific words, such as A / B / C / D must be answered by voice before the voice answering can be carried out. This answering method is very limited, especially for young children, who like to answer questions in various ways such as "the third one", "3", "I choose three", "I choose the third one", etc. If it fails to be triggered many times, it is easy to make users bored and the user experience is poor.

[0046] Therefore, there is an urgent need for a voice answering method that is not restricted by specific trigger words, is more flexible and has high accuracy, so as to improve the user's voice answering experience.

[0047] In view of the above problems, an embodiment of the present invention provides a voice answering method. Figure 1 It is a flow chart of the voice answering method provided by the present invention, such as Figure 1 As shown, the method includes:

[0048] Step 110, obtaining the user's voice answer to the current question.

[0049] Specifically, the current question refers to a question that requires a voice answer, and the current question can be any question extracted from the question library, for example, it can be a math subject question such as "addition and subtraction within ten" or "addition and subtraction within one hundred", or an English subject question, or a Chinese language comprehensive question. The current question can be a multiple-choice question, a true-or-false question, or a fill-in-the-blank question, etc., and the embodiment of the present invention does not specifically limit this.

[0050] A user refers to a person who answers the current question by voice. The user can be one person or multiple people. The user can be a preschooler, a primary school student, or an adult.

[0051] Here, the answering voice is the answering voice for the current question, and the answering voice can be obtained through a sound pickup device, and the sound pickup device here can be a smart phone, a tablet computer, or a voice answering device, etc. After the sound pickup device obtains the answering voice through a microphone array, it can also amplify and reduce noise of the answering voice. In addition, the answering voice can be a voice segment formed after the sound pickup is completed, or it can be a voice stream in the real-time sound pickup process. The embodiment of the present invention does not specifically limit this.

[0052] Step 120, based on the characteristics of the alternative answers to the current question in terms of answer numbers and / or answer content, the voice recognition text of the answering voice is converted into an answer to obtain the user's voice answer.

[0053] Specifically, the characteristics of the answer numbering of the alternative answers to the current question refer to the numbering characteristics corresponding to the alternative options for users to choose from. For example, it can be an alphabetical numbering sequence A / B / C / D, or it can be a digital numbering sequence "first", "second", "third", "fourth", or "1", "2", "3", "4", etc.

[0054] The characteristics of the alternative answers to the current question refer to the characteristics of the answer content of each alternative answer. For example, the answer content characteristics of alternative answer numbers A and B are names of people, and the answer content characteristics of alternative answer numbers C and D are place names; for another example, the answer content characteristics of alternative answer numbers A / B / C / D are time, etc.

[0055] For example, the current topic is: Who was the first emperor of China?

[0056] A: Huangdi B: Yandi C: Ying Zheng D: Liu Bang

[0057] Among them, the answer numbering characteristics refer to A / B / C / D, an alphabetical numbering sequence; the answer content characteristics are "Huangdi", "Yandi", "Ying Zheng" and "Liu Bang", all of which are names.

[0058] Considering that the prior art usually triggers answering questions according to specific vocabulary, if the alternative answer number of the current question is an alphabetical number sequence, and the user's answer voice is a digital number sequence, the answer cannot be triggered, and it may be assumed that the user has no answer or has answered incorrectly; or if the user does not directly answer the number of the current question, but directly answers the answer content of the current question, the answer cannot be triggered, resulting in a poor user experience.

[0059] The voice answering method provided in the embodiment of the present invention first performs voice recognition on the answering voice to obtain a voice recognition text of the answering voice, and then converts the obtained voice recognition text into an answer. The user's voice answer obtained after the answer conversion can trigger the answering more sensitively to improve the user experience.

[0060] The answer conversion here can be performed based on the characteristics of the alternative answers to the current question in terms of answer number and / or answer content. If the voice recognition text contains the answer number, the answer number therein is converted to conform to the numbering format of the alternative answers to the current question; if the voice recognition text contains the answer content, the answer content therein is converted to make it closer to the answer content of the alternative answers to the current question.

[0061] It is understandable that the voice answer obtained by answer conversion may only have the answer number, may only have the answer content, may have both the answer number and the answer content, and of course may have neither the answer number nor the answer content.

[0062] Step 130, determine the answer result based on the voice answer and the standard answer to the current question.

[0063] Specifically, after obtaining the voice answer, the voice answer can be compared with the standard answer to the current question.

[0064] Furthermore, when there is only an answer number in the voice answer, the answer number in the voice answer can be compared with the answer number of the standard answer. When the two are the same, the answer result can be considered correct; when the two are different, the answer result can be considered incorrect.

[0065] When only the answer content exists in the voice answer, the answer content in the voice answer can be compared with the answer content of the standard answer. When the two have the same semantics, the answer result can be considered correct; when the two have different semantics, the answer result can be considered incorrect.

[0066] When a voice answer contains both an answer number and an answer content, the answer can be considered correct only if the answer number and answer content in the voice answer are the same as the answer number and answer content of the standard answer; otherwise, the answer is considered incorrect.

[0067] If the voice answer does not include either an answer number or answer content, the answer is considered incorrect.

[0068] The voice answering method provided by the embodiment of the present invention converts the voice recognition text of the answering voice into an answer based on the characteristics of the answer number and / or answer content of the alternative answers to the current question, thereby obtaining the user's voice answer. The answering method is not limited by specific trigger words, making the user's answering more flexible, and can effectively improve the user experience while ensuring the accuracy of voice answer recognition.

[0069] Based on the above embodiments, Figure 2 is a flow chart of step 120 in the voice answering method provided by the present invention, such as Figure 2 As shown, step 120 specifically includes:

[0070] Step 121, converting the answer number text in the speech recognition text into a number format, and obtaining a speech answer number that conforms to the format of the alternative answer in the answer number;

[0071] And / or, step 122, extracting entities of a preset type from the speech recognition text, and using the extracted entities as speech answer content, wherein the preset type is determined based on the characteristics of the alternative answers in the answer content;

[0072] Step 123, determining the voice answer based on the voice answer number and / or the voice answer content.

[0073] Specifically, the answer number text in the speech recognition text can be extracted based on keyword matching or preset rules. Number format conversion refers to converting the number format corresponding to the answer number text in the speech recognition text into a format that conforms to the answer number of the alternative answer. The voice answer number obtained after format conversion can directly trigger the voice answer.

[0074] For example, the numbering format of the alternative answers is an alphabetical sequence number, more specifically, the four alternative answer numbers A / B / C / D; if the extracted answer number text is "the third one", then "the third one" is converted to "C"; if the extracted answer number text is "the first one", then "the first one" is converted to "A"; if the extracted answer number text is "the fifth one", it is considered that there is no voice answer number in the voice answer.

[0075] Entity extraction for speech recognition text can be achieved through an entity extraction model. The entities referred to here include the answer content contained in the speech recognition text. The speech recognition text can be input into a pre-trained entity extraction model, and the entity extraction model will perform preset entity recognition on the speech recognition text and output the entity annotation of each word in the entity text. The annotation system for entity recognition here can be BIO, BIOES, etc. B indicates the beginning of the entity, E indicates the end of the entity, I indicates the middle word of the entity, O indicates a non-entity, and S indicates a single entity. Among them, the preset type is determined based on the characteristics of the alternative answers in the answer content. For example, the preset type can be a person's name, a place name, a time, or an ancient poem, etc.

[0076] Before executing step 122, the entity extraction model may be trained, wherein the training method of the entity extraction model may include the following steps: first, a large number of sample speech recognition texts are collected, and entities and entity types in the sample speech recognition texts are manually annotated. Then, based on the sample speech recognition texts and the annotated entities and entity types therein, an initial entity extraction model is trained to obtain an entity extraction model.

[0077] The entities extracted by the entity extraction model are used as the voice answer content.

[0078] The voice answer number can be used as the voice answer; or the voice answer content can be used as the voice answer; or both the voice answer number and the voice answer content can be used as the voice answer.

[0079] The method provided by the embodiment of the present invention converts the answer number text into a number format to obtain a voice answer number; extracts a preset type of entity from the voice recognition text to obtain the voice answer content, and determines the voice answer based on the voice answer number and / or the voice answer content. The voice answer thus obtained conforms to the number format and / or content characteristics of the alternative answers, can trigger the answer more sensitively, is not restricted by the triggering of specific words, and makes the user answer more flexible, thereby improving the user experience.

[0080] Based on any of the above embodiments, the preset type is determined based on the entity type of the answer content in the candidate answers.

[0081] Specifically, a preset type of entity extraction is performed on the speech recognition text, and the extracted entity is used as the speech answer content. The preset type is determined based on the entity type of the answer content in the candidate answers.

[0082] The entity type of the answer content in the candidate answers can also be obtained through the entity extraction model. The answer content of each candidate answer is input into the pre-selected and trained entity extraction model, and the entity type extracted by the entity extraction model is used as the preset type.

[0083] Since the entity of the answer content in the candidate answer may be one or more, accordingly, the preset type here may be one or more. When the preset type is multiple, entity extraction of the preset type is performed on the speech recognition text respectively.

[0084] The method provided by the embodiment of the present invention takes the entity type of the answer content in the alternative answers as a preset type, and extracts the preset type of entity from the speech recognition text, thereby obtaining the speech answer content.

[0085] Based on any of the above embodiments, Figure 3 is one of the flow charts of step 130 in the voice answering method provided by the present invention, such as Figure 3 As shown, step 130 specifically includes:

[0086] Step 131, if there is voice answer content in the voice answer, convert the voice answer content into pinyin to be reviewed;

[0087] Step 132, determine the answer result based on the pinyin to be reviewed and the pinyin of the standard answer.

[0088] When there is voice answer content in the voice answer, considering the entity text extracted from the voice recognition text to judge the correctness of the voice answer, there may be a situation where the entity text is the same as the text provided in the actual standard answer but the text shape is different, resulting in incorrect answer recognition, thereby reducing the user experience.

[0089] In order to further improve the accuracy of answer recognition and enhance user experience, the method provided in the embodiment of the present invention converts the voice answer content into the pinyin to be reviewed. It can be understood that the pinyin to be reviewed is the pinyin corresponding to the text of the voice answer content. Here, the answer pinyin of the standard answer refers to the pinyin corresponding to the text of the standard answer content.

[0090] Then compare the pinyin of the answer to be reviewed with the pinyin of the standard answer. If the two are the same, the answer is correct; otherwise, the answer is incorrect.

[0091] Furthermore, considering that the Mandarin pronunciation standards of users who answer questions by voice vary, for example, there may be confusion in the pronunciation of flat and retroflex tongue, and inability to distinguish between l and n in the answering voice, in order to further improve the recognition effect of the answering voice, when converting the standard answer into the standard answer pinyin, when encountering flat and retroflex tongue, l and n, a synchronous expansion conversion will be performed, for example:

[0092] The current topic is: Which poet in Chinese history is called the "Poet Immortal"?

[0093] A Du Fu B Li Bai C Bai Juyi D Shi Hao

[0094] Among them, when converting the pinyin of the standard answer "Li Bai", synchronous expansion will be performed. The pinyin of the standard answer includes two types: "li bai" and "ni bai", so as to be able to recognize the answering voices of users who cannot distinguish between L and N. That is, even if the voice answered by the user is "ni bai", it can still be successfully matched, further improving the user experience.

[0095] In addition, for user groups who are prone to confusing the pronunciation of flat and rolled tongues, the method of synchronous expansion conversion can also be used to improve the recognition accuracy of voice answers. For example, assuming that option D is the standard answer, when converting the pinyin of the content "Shi Hao" in the standard answer, it will include two types: "shi hao" and "si hao". That is, even if the voice answered by the user is "si hao", it can still be successfully matched.

[0096] The method provided by the embodiments of the present invention compares the pinyin of the voice answering content with the pinyin of the standard answer to judge the correctness of the user's voice answering content, which can effectively avoid the situation of homophones with different characters occurring during the voice recognition process of the answering voice, thereby further improving the accuracy of answer recognition and enhancing the user experience.

[0097] In addition, by using the method of synchronous expansion for pinyin conversion of the standard answer, even if the user's Mandarin pronunciation is inaccurate, the pinyin of the standard answer can be accurately matched, thereby improving the accuracy of voice answer recognition and enhancing the user experience.

[0098] Based on any of the above embodiments, the answering voice of the user for the current question can also be directly converted into a pinyin to be reviewed, and based on the pinyin to be reviewed and the answer pinyin of the standard answer, the answering result can be determined.

[0099] Based on any of the above embodiments, Figure 4 is the second schematic flowchart of step 130 in the voice answering method provided by the present invention, as Figure 4 shown, step 130 specifically includes:

[0100] Step 133, based on finger answering and / or sight answering, correct the voice answering to obtain the user's answer. The finger answering is determined based on the finger image of the user pointing to the answering screen, and the sight answering is determined based on the user's face image;

[0101] Step 134, based on the user's answer and the standard answer of the current question, determine the answering result.

[0102] Specifically, in order to further improve the convenience and accuracy of answering, the voice answering can also be corrected based on finger answering and / or sight answering, so as to obtain the corrected user answer.

[0103] The finger answer refers to the answer determined according to the area of ​​the answer screen pointed to by the user's finger. Here, the finger and the answer screen can be in contact or non-contact, which is not specifically limited in the embodiment of the present invention.

[0104] The answer screen refers to the screen of an electronic device used to display the current question. The answer screen can be a touch screen or a non-touch screen.

[0105] The finger image is the finger image of the user pointing to the answering screen, which can be acquired by an image acquisition device. The image acquisition device can be integrated in the electronic device used for answering questions, or it can be an external image acquisition device connected via a universal serial bus (USB).

[0106] The sight line answer refers to the answer determined based on the area of ​​the answer screen where the user's sight line is looking. The sight line answer can be determined based on the user's face image, which contains human eye features. Based on the human eye features, the area of ​​the answer screen where the user's sight line is looking is determined, thereby determining the sight line answer.

[0107] First, the voice answer is corrected by the finger answer and / or the sight answer to obtain the user answer. For example, when the finger answer and / or the sight answer and the voice answer both indicate the same answer, it can be further determined that the recognized voice answer is used as the user answer; when the voice answer is empty and the finger answer and / or the sight answer are obtained, the user can be reminded to make a voice answer in time or be asked whether to use the finger answer and / or the sight answer as the user answer; the result of the finger answer and / or the sight answer can also be directly used as the user answer.

[0108] The answer is then determined based on the user's answer and the standard answer to the current question. The user's answer is compared with the standard answer to determine whether the user's answer is correct.

[0109] Since the user answer comprehensively considers voice answer, finger answer and / or sight answer, the method can effectively improve the accuracy of user answer recognition and further improve the user experience. In addition, finger answer and / or sight answer also make the answer method more flexible.

[0110] The method provided by the embodiment of the present invention determines the answer result by comprehensively considering the user's answers by voice, finger and / or sight, so that the answering method becomes more flexible, effectively improves the accuracy of user answer recognition, and thus further improves the user experience.

[0111] Based on any of the above embodiments, Figure 5 FIG. 1 is a flow chart of the finger answer determination method provided by the present invention. Figure 5 As shown, the finger answer is determined based on the following steps:

[0112] Step 510, obtaining the finger pointing vector of the finger image;

[0113] Step 520, determine the finger answer based on the placement angle of the answer screen, the area occupied by each alternative answer row on the answer screen, and the projection position of the finger pointing amount on the answer screen.

[0114] Specifically, the finger pointing vector is used to characterize the direction in which the finger is pointing to the answer screen. A direction extraction model can be used to extract the finger pointing features in the finger image. The finger pointing features can be output as a result represented by a vector.

[0115] The placement angle of the answering screen refers to the angle between the screen of the electronic answering device and the supporting plane of the electronic device. For example, it is 0 degrees when placed flat, 90 degrees when placed vertically, etc. The placement angle of the answering screen can be flexibly set according to the user's preference.

[0116] The placement angle of the answer screen can be measured by an automated angle measuring instrument. For example, the placement angle of the current answer screen can be displayed and adjusted by the Simply Angle application.

[0117] Taking the current question as an example, a text detection algorithm can be used to segment the text lines of the current question and the images of the alternative answers to obtain the area occupied by each alternative answer on the answer screen. For example, DBNet (Real-time Scene Text Detection with Differentiable Binarization), PSENet or PANNet can be used. The area occupied by each alternative answer on the answer screen can be represented by a rectangular box, which can be specifically represented by the coordinate position of each vertex of the rectangular box on the answer screen.

[0118] After obtaining the placement angle of the answer screen and the area occupied by each candidate answer on the answer screen row, the finger answer can be determined according to the projection position of the finger pointing amount on the answer screen. The candidate answer corresponding to the answer screen area indicated by the projection position is the answer of the finger answer.

[0119] The method provided by the embodiment of the present invention determines the finger answer by the projection position of the finger pointing amount on the answering screen, thereby further improving the accuracy of answer recognition and improving the user experience.

[0120] Based on any of the above embodiments, Figure 6 is a flow chart of the sight line answer determination method provided by the present invention, such as Figure 6As shown, the line of sight answer is determined based on the following steps:

[0121] Step 610, determining the sight direction of the human eye based on the human eye features of the face image;

[0122] Step 620, determining the spatial position of the human eye based on the key point information of the face image;

[0123] Step 630, determining a sight line answer based on the sight line direction and spatial position of the human eye.

[0124] Specifically, during the question answering process, the user can use an image acquisition device, such as a camera or a camera, to acquire the user's face image in real time, or select a video frame containing the user's face from a real-time video as the face image. Human eye features are extracted from the face image to obtain human eye features.

[0125] By inputting the human eye features into the neural network model, the deflection angles of the user's line of sight in the horizontal and vertical directions can be predicted, and the line of sight direction of the human eye in the camera coordinate system can be obtained through post-processing.

[0126] It is understandable that before sending the human eye feature into the neural network model, the human eye feature image can also be normalized, for example, by using a BN (Batch Normalization) algorithm for normalization.

[0127] On the other hand, the key point information of the face image can be used to estimate the spatial position information of the human eye in the camera coordinate system. Specifically, the key point detection model of the face can be used to detect the key points of the human eye in the face area to obtain the coordinate values ​​of the key points of the human eye. In this step, the key point detection model of the face can be implemented by any model of key point detection of the face in the prior art, and this embodiment will not be repeated here.

[0128] It should be noted that steps 610 and 620 can be executed sequentially, but the execution order of the two is not limited; of course, they can also be executed simultaneously.

[0129] Finally, the user's line of sight is determined based on the direction and spatial position of the human eye's line of sight, and then mapped to a pixel point on the answer screen. The user's line of sight answer is judged based on whether the pixel point falls in the area where the alternative answers are located.

[0130] The method provided by the embodiment of the present invention determines the line of sight answer through the human eye features and key point information of the facial image, thereby further improving the accuracy of answer recognition and enhancing the user experience.

[0131] Based on any of the above embodiments, this embodiment provides a voice answering method, the method comprising:

[0132] (1) Obtain the user's voice answer to the current question.

[0133] (2) converting the answer number text in the speech recognition text of the answer voice into a number format to obtain a speech answer number that conforms to the format of the answer number of the alternative answer; and / or performing a preset type of entity extraction on the speech recognition text, and using the extracted entity as the speech answer content, wherein the preset type is determined based on the entity type of the answer content in the alternative answer.

[0134] (3) Determine the voice response based on the voice response number and / or the voice response content.

[0135] (4) Based on the voice response and the standard answer to the current question, determine the answer result and display the user's answer result for the current question.

[0136] (5) When there is voice answer content in the voice answer, the voice answer content is converted into pinyin to be reviewed; based on the pinyin to be reviewed and the pinyin of the standard answer, the answer result is determined, and the user's answer result for the current question is displayed.

[0137] (6) In order to further improve the convenience of answering questions, the method also includes: based on finger answers and / or line of sight answers, the voice answers are corrected to obtain user answers.

[0138] Among them, the finger answer is determined based on the following steps: obtaining the finger pointing vector of the finger image; determining the finger answer based on the placement angle of the answer screen, the area occupied by each alternative answer in the answer screen row, and the projection position of the finger pointing amount on the answer screen.

[0139] The line of sight answer is determined based on the following steps: determining the line of sight direction of the human eye based on the human eye features of the face image; determining the spatial position of the human eye based on the key point information of the face image; determining the line of sight answer based on the line of sight direction and spatial position of the human eye.

[0140] The voice question answering device provided by the present invention is described below. The voice question answering device described below and the voice question answering method described above can be referenced to each other.

[0141] Figure 7 : is a schematic diagram of the structure of the voice answering device provided by the present invention, such as Figure 7 As shown, the device comprises:

[0142] The voice acquisition unit 710 is used to acquire the user's answer voice for the current question;

[0143] The answer conversion unit 720 is used to convert the speech recognition text of the answering speech into an answer based on the characteristics of the candidate answers of the current question in terms of answer numbers and / or answer contents, so as to obtain the voice answer of the user;

[0144] The result determination unit 730 is used to determine the answer result based on the voice answer and the standard answer to the current question.

[0145] The voice answering device provided by the embodiment of the present invention converts the voice recognition text of the answering voice into an answer based on the characteristics of the answer number and / or answer content of the alternative answers to the current question, thereby obtaining the user's voice answer. The answering device is not restricted by specific trigger words, making the user's answering more flexible, and can effectively improve the user experience while ensuring the accuracy of voice answer recognition.

[0146] Based on any of the above embodiments, the answer conversion unit 720 is further used for:

[0147] Convert the answer number text in the speech recognition text into a number format to obtain a speech answer number that conforms to the format of the answer number of the alternative answer;

[0148] and / or, extracting entities of a preset type from the speech recognition text, and using the extracted entities as speech answer content, wherein the preset type is determined based on the characteristics of the alternative answers in terms of the answer content;

[0149] The voice answer is determined based on the voice answer number and / or the voice answer content.

[0150] Based on any of the above embodiments, the preset type is determined based on the entity type of the answer content in the candidate answers.

[0151] Based on any of the above embodiments, the result determination unit 730 is further configured to:

[0152] If there is voice answer content in the voice answer, convert the voice answer content into pinyin to be reviewed;

[0153] The answer result is determined based on the pinyin to be reviewed and the pinyin of the standard answer.

[0154] Based on any of the above embodiments, the result determination unit 730 is further configured to:

[0155] Based on the finger answer and / or the sight answer, the voice answer is modified to obtain the user answer, wherein the finger answer is determined based on the finger image of the user pointing to the answer screen, and the sight answer is determined based on the face image of the user;

[0156] The answer result is determined based on the user's answer and the standard answer to the current question.

[0157] Based on any of the above embodiments, the voice answering device provided by the embodiment of the present invention further includes a finger answer determining unit, which is used to:

[0158] Obtaining a finger pointing vector of the finger image;

[0159] The finger answer is determined based on the placement angle of the answer screen, the area occupied by each alternative answer row on the answer screen, and the projection position of the finger pointing amount on the answer screen.

[0160] Based on any of the above embodiments, the voice answering device provided by the embodiment of the present invention further includes a sight answer determining unit, which is used to:

[0161] Determining the sight direction of the human eye based on the human eye features of the facial image;

[0162] Determining the spatial position of the human eye based on the key point information of the facial image;

[0163] The gaze answer is determined based on the gaze direction and spatial position of the human eye.

[0164] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the voice answering method, which includes: obtaining the user's answering voice for the current question; based on the characteristics of the alternative answers of the current question in terms of answer number and / or answer content, converting the voice recognition text of the answering voice to obtain the user's voice answer; and determining the answering result based on the voice answer and the standard answer of the current question.

[0165] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0166] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the voice answering method provided by the above methods, which includes: obtaining the user's answer voice to the current question; based on the characteristics of the alternative answers to the current question in terms of answer number and / or answer content, converting the voice recognition text of the answer voice to obtain the user's voice answer; based on the voice answer and the standard answer to the current question, determining the answer result.

[0167] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the voice answering method provided by the above-mentioned methods, the method comprising: obtaining the user's answer voice to the current question; based on the characteristics of the alternative answers to the current question in terms of answer number and / or answer content, performing answer conversion on the voice recognition text of the answer voice to obtain the user's voice answer; based on the voice answer and the standard answer to the current question, determining the answer result.

[0168] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0169] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A voice answering method, It is characterized in that include: Get the user's answer to the current question; Based on the characteristics of the alternative answers to the current question in terms of answer number and / or answer content, the voice recognition text of the answer voice is converted into an answer to obtain the user's voice answer; if the voice recognition text contains an answer number, the answer number is converted to match the numbering format of the alternative answers to the current question; if the voice recognition text contains answer content, the answer content is converted to match the answer content of the alternative answers to the current question; based on the voice answer and the standard answer to the current question, the answer result is determined.

2. The voice answering method according to claim 1, It is characterized in that The step of converting the voice recognition text of the answering voice into an answer based on the characteristics of the alternative answers of the current question in terms of answer numbers and / or answer contents to obtain the voice answer of the user includes: Convert the answer number text in the speech recognition text into a number format to obtain a speech answer number that conforms to the format of the answer number of the alternative answer; and / or, extracting entities of a preset type from the speech recognition text, and using the extracted entities as speech answer content, wherein the preset type is determined based on the characteristics of the alternative answers in terms of the answer content; The voice answer is determined based on the voice answer number and / or the voice answer content.

3. The voice answering method according to claim 2, It is characterized in that The preset type is determined based on the entity type of the answer content in the candidate answers.

4. The voice answering method according to claim 1, It is characterized in that The step of determining the answer result based on the voice answer and the standard answer to the current question includes: If there is voice answer content in the voice answer, convert the voice answer content into pinyin to be reviewed; The answer result is determined based on the pinyin to be reviewed and the pinyin of the standard answer.

5. The voice answering method according to any one of claims 1 to 4, It is characterized in that The step of determining the answer result based on the voice answer and the standard answer to the current question includes: Based on the finger answer and / or the sight answer, the voice answer is modified to obtain the user answer, wherein the finger answer is determined based on the finger image of the user pointing to the answer screen, and the sight answer is determined based on the face image of the user; The answer result is determined based on the user's answer and the standard answer to the current question.

6. The voice answering method according to claim 5, It is characterized in that The finger answer is determined based on the following steps: Obtaining a finger pointing vector of the finger image; The finger answer is determined based on the placement angle of the answer screen, the area occupied by each alternative answer row on the answer screen, and the projection position of the finger pointing amount on the answer screen.

7. The voice answering method according to claim 5, It is characterized in that The line of sight answer is determined based on the following steps: Determining the sight direction of the human eye based on the human eye features of the facial image; Determining the spatial position of the human eye based on the key point information of the facial image; The gaze answer is determined based on the gaze direction and spatial position of the human eye.

8. A voice answering device, It is characterized in that include: A voice acquisition unit, used to acquire the user's answer voice for the current question; The answer conversion unit is used to convert the voice recognition text of the answer voice into an answer based on the characteristics of the alternative answers of the current question in terms of answer numbers and / or answer contents, so as to obtain the voice answer of the user; if the voice recognition text contains an answer number, the answer number is converted to conform to the numbering format of the alternative answers of the current question; if the voice recognition text contains an answer content, the answer content is converted to conform to the answer content of the alternative answers of the current question; The result determination unit is used to determine the answer result based on the voice answer and the standard answer to the current question.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the voice answering method as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the voice answering method as described in any one of claims 1 to 7 is implemented.

Images