Question type identification method and device based on multi-modal model, medium and equipment

Abstract

The invention discloses a question type recognition method and device based on a multi-modal model, a medium and equipment, and the method comprises the steps: obtaining a question image of a to-be-recognized question, carrying out the character recognition of the question image, and determining a question text corresponding to the question image; performing fusion identification on the question image and the question text through a multi-modal model, and determining a multi-modal identification result of the question to be identified; matching the question image according to a question image library, determining an image recognition result of the question to be recognized, and / or matching the question text according to a question text library, and determining a text recognition result of the question to be recognized; according to the multi-modal recognition result and a single-modal recognition result, the question type of the question to be recognized is determined, and the single-modal recognition result is the text recognition result and / or the image recognition result. And the accuracy of question type identification can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a question type recognition method, device, medium and equipment based on a multimodal model. Background Art

[0002] In the automatic grading system, accurate identification of question types is crucial for choosing the appropriate grading method. However, the diversity of question types poses a huge challenge to recognition. Traditional recognition strategies mainly rely on text content-based analysis. This method performs well when dealing with questions with rich text descriptions, but its recognition ability drops significantly when faced with illustration questions with less text, and it is often unable to accurately determine the question type. What is particularly tricky is that in the field of calculation questions, traditional recognition methods have difficulty effectively distinguishing fine-grained question types such as horizontal, vertical, and freehand, and often make misjudgments.

[0003] On the other hand, although the attempt to use image-based detection and classification methods has improved the recognition ability of questions with rich visual elements to a certain extent, when faced with questions with more text, the limitations of image features lead to a significant reduction in recognition accuracy, and the problem of misrecognition still exists.

[0004] Currently, neither text-based nor image-based methods can fully cope with complex and changing question types. Therefore, how to efficiently and accurately identify various types of questions has become an urgent problem to be solved. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a method, apparatus, medium and equipment for question type recognition based on a multimodal model. By fusing information from both image and text modalities, the multimodal model can more comprehensively capture the characteristics of the question, thereby improving recognition accuracy. By matching and identifying the question image library and the question text library, the multimodal recognition results can be further verified and confirmed, thereby reducing the possibility of misrecognition and improving the degree of automation of the grading system and its ability to support fine-grained recognition.

[0006] According to one aspect of the present application, a question type identification method based on a multimodal model is provided, the method comprising:

[0007] Acquire a question image of a question to be identified, and perform text recognition on the question image to determine the question text corresponding to the question image;

[0008] Performing fusion recognition on the question image and the question text through a multimodal model to determine a multimodal recognition result of the question to be recognized;

[0009] Matching the question image according to the question image library to determine the image recognition result of the question to be recognized, and / or matching the question text according to the question text library to determine the text recognition result of the question to be recognized;

[0010] The question type of the question to be recognized is determined according to the multimodal recognition result and the unimodal recognition result, wherein the unimodal recognition result is the text recognition result and / or the image recognition result.

[0011] In an embodiment of the present application, optionally, the multimodal model includes an image feature extraction module, a text feature extraction module, a word frequency feature extraction module, and a question type classification module; the multimodal model is used to perform fusion recognition on the question image and the question text to determine the multimodal recognition result of the question to be recognized, including:

[0012] Extracting image features corresponding to the title image by the image feature extraction module;

[0013] Perform word segmentation processing on the topic text by the text feature extraction module, and extract text features corresponding to the topic text based on the word segmentation result;

[0014] Extracting the word frequency feature corresponding to the title text according to the word segmentation result by the word frequency feature extraction module;

[0015] The question type classification module identifies the question type of the question to be identified based on the image features, the text features and the word frequency features to determine the confidence that the question to be identified belongs to each preset question type as the multimodal recognition result.

[0016] In the embodiment of the present application, optionally, extracting text features corresponding to the title text based on the word segmentation result includes:

[0017] The text feature extraction module removes stop words from the segmented words obtained by the segmentation process to obtain a segmentation result;

[0018] For any word segmentation in the word segmentation result, extract the word feature of the word segmentation, obtain the number of occurrences of the word segmentation in each preset question type in the preset word statistical table, generate the question type weight feature corresponding to the word segmentation according to the number of occurrences of the word segmentation in each preset question type, and construct the question type weight feature and the word feature into the word segmentation feature of the word segmentation, wherein the preset word statistical table includes the number of occurrences of multiple preset words corresponding to each preset question type;

[0019] The text features corresponding to the title text are generated according to the segmentation features of each segmentation word.

[0020] In an embodiment of the present application, optionally, the method for constructing the preset word statistics table includes:

[0021] Obtaining a sample set of questions, wherein the sample set of questions includes a plurality of sample questions corresponding to each preset question type;

[0022] Perform word segmentation and stop word removal on each question sample to obtain word segmentation information in each question sample;

[0023] Each segmentation word in the segmentation information is used as the preset word, the number of occurrences of the same preset word in each preset question type is counted, and the preset word statistics table is constructed.

[0024] In the embodiment of the present application, optionally, extracting the word frequency feature corresponding to the title text according to the word segmentation result by the word frequency feature extraction module includes:

[0025] For any segmentation in the segmentation results, according to the number of occurrences of the segmentation in each preset question type and the total number of occurrences of the segmentation in each preset question type, the word frequency of the segmentation in each preset question type is calculated respectively, and the word frequency of the segmentation in each preset question type is normalized;

[0026] The normalized word frequencies corresponding to each word segment belonging to the same preset question type are summed up to determine the summed word frequencies corresponding to each preset question type, and the word frequency features of the question text are constructed according to the summed word frequencies corresponding to each preset question type.

[0027] In an embodiment of the present application, optionally, the text feature extraction module removes stop words from the segmented words obtained by the segmentation processing to obtain a segmentation result, including: the text feature extraction module removes stop words from the segmented words obtained by the segmentation processing based on the preset word statistics table to obtain a segmentation result; and / or, the word frequency of the segmented words in each preset question type is normalized, including: calculating the total word frequency of the segmented words in each preset question type, adding the total word frequency to a preset value to obtain a new total word frequency, and respectively calculating the ratio of the word frequency of the segmented words in each preset question type to the new total word frequency as the normalized word frequency of the segmented words in each preset question type.

[0028] In the embodiment of the present application, optionally, matching the title image according to the title image library to determine the image recognition result of the title to be recognized includes:

[0029] Acquire the image features extracted by the image feature extraction module;

[0030] Calculate the image similarity between the image feature and each image feature sample in the question image library, sort the image feature samples from large to small in image similarity, and take a preset first number of image feature samples sorted first to construct the image recognition result, wherein the image recognition result includes the first question type information and image similarity corresponding to the preset first number of image feature samples.

[0031] In the embodiment of the present application, optionally, matching the question text according to the question text library to determine the text recognition result of the question to be recognized includes:

[0032] Calculate the text similarity between the question text and each question text sample in the question text library, sort the question text samples from large to small in text similarity, and take a preset second number of question text samples ranked first to construct the text recognition result, wherein the text recognition result includes the second question type information and text similarity corresponding to the preset second number of question text samples.

[0033] In an embodiment of the present application, optionally, the multimodal recognition result includes the confidence that the question to be recognized belongs to each preset question type; the image recognition result includes the first question type information and image similarity corresponding to a preset first number of image feature samples; the text recognition result includes the second question type information and text similarity corresponding to a preset second number of question text samples;

[0034] Determining the type of the question to be identified according to the multimodal recognition result and the unimodal recognition result includes:

[0035] Traversing the image recognition results, and adjusting the confidence corresponding to the first question type information of the currently traversed image feature sample in the confidence of each preset question type according to the image similarity of the currently traversed image feature sample, so as to update the confidence of each preset question type; and / or traversing the text recognition results, and adjusting the confidence corresponding to the second question type information of the currently traversed question text sample in the confidence of each preset question type according to the text similarity of each traversed question text sample, so as to update the confidence of each preset question type;

[0036] The question type of the question to be identified is determined according to the updated confidence levels of the preset question types.

[0037] In the embodiment of the present application, optionally, adjusting the confidence corresponding to the first question type information of the currently traversed image feature sample among the confidences of each preset question type according to the image similarity of the currently traversed image feature sample includes:

[0038] If the image similarity of the currently traversed image feature sample is greater than or equal to the preset image similarity, then based on the preset first increment value, the confidence of each preset question type corresponding to the first question type information of the currently traversed image feature sample is increased and adjusted, otherwise the traversal of the image recognition result is terminated;

[0039] The step of adjusting the confidence of each preset question type corresponding to the second question type information of the currently traversed question text sample according to the text similarity of each traversed question text sample to update the confidence of each preset question type includes:

[0040] If the text similarity of the currently traversed question text sample is greater than or equal to the preset text similarity, then based on the preset second incremental value, the confidence of each preset question type corresponding to the second question type information of the currently traversed question text sample is increased and adjusted, otherwise the traversal of the text recognition result is terminated.

[0041] In the embodiment of the present application, optionally, before increasing and adjusting the confidence corresponding to the first question type information of the currently traversed image feature sample among the confidences of the preset question types based on the preset first increment value, the method further includes:

[0042] If the image similarity of the currently traversed image feature sample is greater than or equal to a preset image similarity, then according to a target image similarity interval among a plurality of preset image similarity intervals into which the image similarity of the currently traversed image feature sample falls, a preset first increment value corresponding to the target similarity interval is determined, wherein the preset image similarity is a minimum value of a plurality of preset image similarity intervals;

[0043] Before increasing and adjusting the confidence of each preset question type corresponding to the second question type information of the currently traversed question text sample based on the preset second increment value, the method further includes:

[0044] If the text similarity of the currently traversed question text sample is greater than or equal to the preset text similarity, then according to the target text similarity interval among the multiple preset text similarity intervals into which the text similarity of the currently traversed question text sample falls, a preset second incremental value corresponding to the target text similarity interval is determined, wherein the preset text similarity is the minimum value of the multiple preset text similarity intervals.

[0045] According to another aspect of the present application, a question type identification device based on a multimodal model is provided, the device comprising:

[0046] A text recognition module is used to obtain a question image of a question to be recognized, and perform text recognition on the question image to determine the question text corresponding to the question image;

[0047] A multimodal recognition module, used to perform fusion recognition on the question image and the question text through a multimodal model to determine a multimodal recognition result of the question to be recognized;

[0048] A single-mode recognition module is used to match the question image according to the question image library to determine the image recognition result of the question to be recognized, and / or to match the question text according to the question text library to determine the text recognition result of the question to be recognized;

[0049] A question type determination module is used to determine the question type of the question to be identified based on the multimodal recognition result and the unimodal recognition result, wherein the unimodal recognition result is the text recognition result and / or the image recognition result.

[0050] In an embodiment of the present application, optionally, the multimodal model includes an image feature extraction module, a text feature extraction module, a word frequency feature extraction module, and a question type classification module; the multimodal recognition module is further used to:

[0051] Extracting image features corresponding to the title image by the image feature extraction module;

[0052] Perform word segmentation processing on the topic text by the text feature extraction module, and extract text features corresponding to the topic text based on the word segmentation result;

[0053] Extracting the word frequency feature corresponding to the title text according to the word segmentation result by the word frequency feature extraction module;

[0054] The question type classification module identifies the question type of the question to be identified based on the image features, the text features and the word frequency features to determine the confidence that the question to be identified belongs to each preset question type as the multimodal recognition result.

[0055] In the embodiment of the present application, optionally, the multimodal recognition module is further used to:

[0056] The text feature extraction module removes stop words from the segmented words obtained by the segmentation process to obtain a segmentation result;

[0057] For any word segmentation in the word segmentation result, extract the word feature of the word segmentation, obtain the number of occurrences of the word segmentation in each preset question type in the preset word statistical table, generate the question type weight feature corresponding to the word segmentation according to the number of occurrences of the word segmentation in each preset question type, and construct the question type weight feature and the word feature into the word segmentation feature of the word segmentation, wherein the preset word statistical table includes the number of occurrences of multiple preset words corresponding to each preset question type;

[0058] The text features corresponding to the title text are generated according to the segmentation features of each segmentation word.

[0059] In the embodiment of the present application, optionally, the device further includes: a statistical table construction module, which is used to:

[0060] Obtaining a sample set of questions, wherein the sample set of questions includes a plurality of sample questions corresponding to each preset question type;

[0061] Perform word segmentation and stop word removal on each question sample to obtain word segmentation information in each question sample;

[0062] Each segmentation word in the segmentation information is used as the preset word, the number of occurrences of the same preset word in each preset question type is counted, and the preset word statistics table is constructed.

[0063] In the embodiment of the present application, optionally, the multimodal recognition module is further used to:

[0064] For any segmentation in the segmentation results, according to the number of occurrences of the segmentation in each preset question type and the total number of occurrences of the segmentation in each preset question type, the word frequency of the segmentation in each preset question type is calculated respectively, and the word frequency of the segmentation in each preset question type is normalized;

[0065] The normalized word frequencies corresponding to each word segment belonging to the same preset question type are summed up to determine the summed word frequencies corresponding to each preset question type, and the word frequency features of the question text are constructed according to the summed word frequencies corresponding to each preset question type.

[0066] In the embodiment of the present application, optionally, the multimodal recognition module is further used to:

[0067] The text feature extraction module removes stop words from the segmented words obtained by the segmentation process based on the preset word statistics table to obtain a segmentation result; and / or,

[0068] Calculate the total word frequency of the segmented word in each preset question type, add the total word frequency to the preset value to obtain a new total word frequency, and calculate the ratio of the word frequency of the segmented word in each preset question type to the new total word frequency as the normalized word frequency of the segmented word in each preset question type.

[0069] In the embodiment of the present application, optionally, the single-mode recognition module is further used to:

[0070] Acquire the image features extracted by the image feature extraction module;

[0071] Calculate the image similarity between the image feature and each image feature sample in the question image library, sort the image feature samples from large to small in image similarity, and take a preset first number of image feature samples sorted first to construct the image recognition result, wherein the image recognition result includes the first question type information and image similarity corresponding to the preset first number of image feature samples.

[0072] In the embodiment of the present application, optionally, the single-mode recognition module is further used to:

[0073] Calculate the text similarity between the question text and each question text sample in the question text library, sort the question text samples from large to small in text similarity, and take a preset second number of question text samples ranked first to construct the text recognition result, wherein the text recognition result includes the second question type information and text similarity corresponding to the preset second number of question text samples.

[0074] In an embodiment of the present application, optionally, the multimodal recognition result includes the confidence that the question to be recognized belongs to each preset question type; the image recognition result includes the first question type information and image similarity corresponding to a preset first number of image feature samples; the text recognition result includes the second question type information and text similarity corresponding to a preset second number of question text samples;

[0075] The question type determination module is further used for:

[0076] Traversing the image recognition results, and adjusting the confidence corresponding to the first question type information of the currently traversed image feature sample in the confidence of each preset question type according to the image similarity of the currently traversed image feature sample, so as to update the confidence of each preset question type; and / or traversing the text recognition results, and adjusting the confidence corresponding to the second question type information of the currently traversed question text sample in the confidence of each preset question type according to the text similarity of each traversed question text sample, so as to update the confidence of each preset question type;

[0077] The question type of the question to be identified is determined according to the updated confidence levels of the preset question types.

[0078] In the embodiment of the present application, optionally, the question type determination module is further used to:

[0079] If the image similarity of the currently traversed image feature sample is greater than or equal to the preset image similarity, then based on the preset first increment value, the confidence of each preset question type corresponding to the first question type information of the currently traversed image feature sample is increased and adjusted, otherwise the traversal of the image recognition result is terminated;

[0080] If the text similarity of the currently traversed question text sample is greater than or equal to the preset text similarity, then based on the preset second incremental value, the confidence of each preset question type corresponding to the second question type information of the currently traversed question text sample is increased and adjusted, otherwise the traversal of the text recognition result is terminated.

[0081] In the embodiment of the present application, optionally, the question type determination module is further used to:

[0082] If the image similarity of the currently traversed image feature sample is greater than or equal to a preset image similarity, then according to a target image similarity interval among a plurality of preset image similarity intervals into which the image similarity of the currently traversed image feature sample falls, a preset first increment value corresponding to the target similarity interval is determined, wherein the preset image similarity is a minimum value of a plurality of preset image similarity intervals;

[0083] If the text similarity of the currently traversed question text sample is greater than or equal to the preset text similarity, then according to the target text similarity interval among the multiple preset text similarity intervals into which the text similarity of the currently traversed question text sample falls, a preset second incremental value corresponding to the target text similarity interval is determined, wherein the preset text similarity is the minimum value of the multiple preset text similarity intervals.

[0084] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned question type recognition method based on the multimodal model is implemented.

[0085] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned multimodal model-based question type recognition method when executing the program.

[0086] By means of the above technical solution, the embodiment of the present application provides a method, device, medium and equipment for question type recognition based on a multimodal model, which performs text recognition on a question image to determine the question text of the question image, performs fusion recognition through a multimodal model, and combines the matching results of the question image library and the question text library to determine the specific question type of the question to be identified. By fusing the information of the two modes of image and text, the multimodal model of the present application can capture the characteristics of the question more comprehensively, thereby improving the accuracy of recognition. Through the matching recognition of the question image library and the question text library, the multimodal recognition results can be further verified and confirmed, thereby reducing the possibility of misrecognition and improving the automation level of the grading system and the ability to support fine-grained recognition.

[0087] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0089] Figure 1A A schematic diagram of a process flow of a question type identification method based on a multimodal model provided in an embodiment of the present application is shown;

[0090] Figure 1B A schematic diagram of a process flow of another question type identification method based on a multimodal model provided in an embodiment of the present application is shown;

[0091] Figure 2A A schematic diagram of a process flow of another question type identification method based on a multimodal model provided in an embodiment of the present application is shown;

[0092] Figure 2B A schematic diagram of the model structure of a multimodal model provided in an embodiment of the present application is shown;

[0093] Figure 3 A schematic structural diagram of a question type recognition device based on a multimodal model provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0094] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0095] In this embodiment, a question type recognition method based on a multimodal model is provided. Figure 1A As shown, the method includes:

[0096] Step 101: Obtain a question image of a question to be identified, and perform text recognition on the question image to determine the question text corresponding to the question image.

[0097] Step 102: Perform fusion recognition on the question image and the question text through a multimodal model to determine a multimodal recognition result of the question to be recognized.

[0098] Step 103: Match the question image according to the question image library to determine the image recognition result of the question to be recognized, and / or match the question text according to the question text library to determine the text recognition result of the question to be recognized.

[0099] Step 104: determine the type of the question to be recognized based on the multimodal recognition result and the unimodal recognition result, wherein the unimodal recognition result is the text recognition result and / or the image recognition result.

[0100] In the embodiment of the present application, first, the image of the question to be identified is obtained, for example, by scanning a paper test paper, taking a photo or intercepting from an electronic document. Then, the text in the question image is identified using optical character recognition (OCR) technology and converted into an editable and processable text format. Secondly, after obtaining the question image and the corresponding question text, the system uses a multimodal model to fuse and identify these two types of information. A multimodal model is a machine learning model that can process multiple types of data (such as images and text) at the same time. It improves the accuracy and robustness of recognition by learning the association and complementarity between images and texts. In this step, the multimodal model will comprehensively consider the visual features in the question image and the text information in the question text, so as to obtain a more accurate and comprehensive recognition result. Since the multimodal model can comprehensively consider a variety of information, it can support more fine-grained recognition of the question type, such as distinguishing horizontal, vertical and detached forms in calculation questions. Then, in order to improve the accuracy and reliability of recognition, the question image library and the question text library can also be used for matching and recognition. The question image library contains a large number of question images of known types, while the question text library contains text descriptions corresponding to these images. By matching these question images and texts to be identified with the data in the library, the multimodal recognition results can be further verified and confirmed based on the image recognition results and text recognition results, thereby reducing the possibility of misrecognition. Finally, the multimodal recognition results and the unimodal recognition results (i.e., the image recognition results and the text recognition results) are combined to determine the specific question type of the question to be identified. Since the multimodal model can comprehensively consider the information of the image and text, its recognition results are usually more accurate and reliable than the unimodal recognition results. However, in some cases, the unimodal recognition results may also provide useful supplementary information. Therefore, the multimodal recognition results can be adjusted based on the unimodal recognition results in order to achieve the effect of combining the two for comprehensive question type recognition to obtain the final question type recognition result. Figure 1BThe figure shows a flowchart of question type recognition in single-modal recognition, which includes question image matching and question text matching. Input the question image of the question to be recognized, and obtain the text content of the question through the OCR module; input the text content and image data into the trained multimodal model for forward reasoning, and output the question type and the corresponding confidence as the multimodal recognition result; use the text content to perform text search in the question text library to obtain the text recognition result; use the question image to perform image search in the question image library to obtain the image recognition result; use the multimodal model recognition result, text recognition result, and image recognition result to fuse and output the final question type recognition result.

[0101] By applying the technical solution of this embodiment, text recognition is performed on the question image to determine the question text of the question image, and fusion recognition is performed through a multimodal model, and the specific question type of the question to be recognized is determined by combining the matching results of the question image library and the question text library. By fusing the information of the two modes of image and text, the embodiment of the present application can more comprehensively capture the characteristics of the question, thereby improving the accuracy of recognition. Through the matching recognition of the question image library and the question text library, the multimodal recognition results can be further verified and confirmed, thereby reducing the possibility of misrecognition and improving the automation level of the grading system and the ability to support fine-grained recognition.

[0102] Furthermore, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, another question type recognition method based on a multimodal model is provided, such as Figure 2A As shown, the method includes:

[0103] Step 201: Obtain a question image of a question to be identified, and perform text recognition on the question image to determine the question text corresponding to the question image.

[0104] Step 202: extract the image features corresponding to the question image through the image feature extraction module; perform word segmentation on the question text through the text feature extraction module, and extract the text features corresponding to the question text based on the word segmentation result; extract the word frequency features corresponding to the question text according to the word segmentation result through the word frequency feature extraction module; identify the question type of the question to be identified according to the image features, the text features and the word frequency features through the question type classification module, so as to determine the confidence that the question to be identified belongs to each preset question type as the multimodal recognition result.

[0105] In this embodiment, if Figure 2BAs shown in the figure, the multimodal model is mainly composed of four core modules: image feature extraction module, text feature extraction module, word frequency feature extraction module, and question type classification module. The image feature extraction module extracts key visual features from the question image to obtain the image features of the question image. Specifically, the question image data is input into the image feature extraction module, and the question image data is preprocessed, such as scaling to a fixed size, image data standardization, etc. The preprocessed image data is further input into the trained image feature extraction model to output a feature vector of fixed dimension. For example, the MobileNetV3 small model is used as the image feature extraction model, and other models can also be used in actual use. The text feature extraction module first performs word segmentation on the text, that is, cutting the continuous text string into independent vocabulary units. On this basis, the module will further extract the semantic features of these words, such as word vectors. The text features of the question text are determined based on the word segmentation results to provide key information for understanding the text content. The word frequency feature extraction module can also combine the word frequency and other information corresponding to each word segmentation in different preset question types according to the word segmentation result determined by the text feature extraction module (or independently perform word segmentation to determine the word segmentation result), and comprehensively determine the word frequency features corresponding to the question text. Finally, the question type classification module will classify the questions using a machine learning algorithm (such as a support vector machine SVM, a random forest RF or a deep learning model) based on the word segmentation result determined by the text feature extraction module. This module will calculate the confidence that the question to be identified belongs to each preset question type, that is, how likely the question is to belong to a specific type. Ultimately, the multimodal recognition results will be presented in the form of these confidences. The embodiment of the present application fuses the information of the two modes of image and text, and combines the word frequency features. The multimodal model can capture the characteristics of the question more comprehensively, thereby improving the accuracy of recognition. The multimodal model can comprehensively consider a variety of information, so it supports more fine-grained recognition of the question type, such as distinguishing different types of calculation questions, application questions, etc. And by calculating the confidence that the question to be identified belongs to each preset question type, the model can provide richer information support for subsequent processing and decision-making.

[0106] In an embodiment of the present application, optionally, the step 202 of extracting text features corresponding to the question text based on the word segmentation results includes: the text feature extraction module removes stop words from the word segmentation results to obtain a word segmentation result; for any word segmentation in the word segmentation result, extracts the word features of the word segmentation, obtains the number of occurrences of the word segmentation in each preset question type in a preset word statistics table, generates a question type weight feature corresponding to the word segmentation according to the number of occurrences of the word segmentation in each preset question type, constructs the question type weight feature and the word feature into the word segmentation feature of the word segmentation, wherein the preset word statistics table includes the number of occurrences of multiple preset words corresponding to each preset question type; and generates the text features corresponding to the question text according to the word segmentation features of each word segmentation.

[0107] In the above embodiments, the text feature extraction module performs stop word removal on the word segments obtained by word segmentation. Stop words refer to those words that frequently appear in the text but do not substantially help in understanding the text content, such as "de", "le", "zai", etc. By removing these stop words, redundant information can be reduced, making subsequent feature extraction more efficient and accurate. Next, for any word segment in the word segment result after stop word removal, the following processing is performed: Convert the word segment into a word vector or obtain its semantic features using other word embedding techniques (such as Word2Vec, BERT, etc.) to achieve the extraction of word features, and reflect the semantic information of the word segment through the word features. Look up the number of occurrences of this word segment in each preset question type in the preset word statistics table. This preset word statistics table is a pre-constructed database that records the statistical information of multiple preset words in different question types. According to the number of occurrences of the word segment in each preset question type, the question type weight feature corresponding to the word segment can be generated. The question type weight feature can reflect the degree of association between the word segment and different question types, providing data support for subsequent question type classification. Finally, combine the question type weight feature and the word feature to construct the word segment feature of the word segment. This word segment feature contains both the semantic information of the word segment and the association information between it and different question types. Further, after obtaining the word segment features of each word segment, integrate these word segment features to generate the text feature corresponding to the question text. This text feature is a feature vector that synthesizes all the word segment information in the question text and can comprehensively reflect the content features and question type association information of the question text. In a specific application scenario, the remaining word segments after stop word removal can be converted into a feature vector with a fixed dimension as the word feature using the text feature extraction module. For example, use word2vec to convert the word segments into word vectors, and then extract a question type weight feature with a fixed dimension for each word. For example, according to the number of occurrences of each word segment in each preset question type in the preset word statistics table, perform normalization to obtain the question type weight vector (i.e., the question type weight feature), and the dimension of the question type weight vector is equal to the number of preset question types. Concatenate the question type weight feature and the word feature into a new vector with a fixed dimension, concatenate the new vectors of each word segment into a matrix, fill the matrix data with fixed values, obtain a matrix data with a fixed size, and then use the text feature extraction module to convert the matrix data into a feature vector with a fixed dimension. In the embodiments of this application, by using the association information between word segments and different question types, the model can better adapt to different types and formats of questions, enhance the generalization ability of the model, and by combining word features and question type weight features, the model can more accurately identify the fine-grained type to which the question belongs, such as distinguishing different types of reading comprehension questions or math application questions, etc.

[0108] In an embodiment of the present application, optionally, the method for constructing the preset word statistics table includes: obtaining a question sample set, wherein the question sample set includes multiple question samples corresponding to each preset question type; performing word segmentation and stop word removal processing on each question sample to obtain word segmentation information in each question sample; using each word in the word segmentation information as the preset word, counting the number of occurrences of the same preset word in each preset question type, and constructing the preset word statistics table.

[0109] In this embodiment, a question sample set containing multiple question samples is obtained, and the sample set contains multiple question samples of each preset question type to ensure that the diversity of various question types and question contents can be captured, and the number of occurrences of each preset word in different question types can be accurately counted. Next, each question sample is segmented and stop word removed, and continuous text strings are cut into independent vocabulary units, and those words that are not substantially helpful for understanding the text content are removed. After processing all question samples, each segmentation in these segmentation information is used as a preset word, and the number of occurrences of the same preset word in each preset question type is counted. Specifically, the entire question sample set can be traversed, and the number of occurrences of each segmentation in each question type is accumulated to construct a comprehensive statistical table. Finally, a preset word statistical table is constructed based on the statistical results. This table is usually a two-dimensional array or database table, which contains multiple preset words and their number of occurrences in different question types. This statistical table is an important basis for subsequent text feature extraction and question type classification tasks, helping the multimodal model to capture the association information between segmentation and different question types.

[0110] In an embodiment of the present application, optionally, the extracting of the word frequency feature corresponding to the question text according to the word segmentation result by the word frequency feature extraction module in step 202 includes: for any word segmentation result, according to the number of occurrences of the word segmentation in each preset question type and the total number of occurrences of the word segmentation in each preset question type, respectively calculating the word frequency of the word segmentation in each preset question type, and normalizing the word frequency of the word segmentation in each preset question type; summing the normalized word frequencies corresponding to each word segmentation belonging to the same preset question type, determining the summed word frequency result corresponding to each preset question type, and constructing the word frequency feature of the question text according to the summed word frequency result corresponding to each preset question type.

[0111] In this embodiment, in the word frequency feature extraction module, for any word in the word segmentation result, the number of occurrences of the word in each preset question type is obtained in the preset word statistics table, and the total number of occurrences of the word in each preset question type is obtained by adding the number of occurrences of the word in each question type, thereby calculating the word frequency of the word in each preset question type. In order to eliminate the deviation caused by the large difference in the number of occurrences between different word segments, the word frequency of the word segmentation in each preset question type can also be normalized. After the normalization process is completed, for each preset question type, the normalized word frequencies corresponding to each word segmentation belonging to the question type are summed, and this summation result reflects the overall word frequency characteristics of all word segmentations in the question type. Finally, the word frequency characteristics of the question text are constructed according to the word frequency summation results corresponding to each preset question type. In a specific application scenario, assuming that the number of occurrences of word segmentation W in question type T is pT, and the number of occurrences of word segmentation W in all preset question types is pA, the word frequency of word segmentation W in question type T can be calculated based on the formula P=pT / (pA-pT+1). The frequency of each word segment in each preset question type is further normalized. After normalization, the normalized word frequencies of each word segment in the same preset question type are added together to obtain the frequency sum of the non-stop words in the current question in the question type. Assuming that the current question has N non-stop words, and their normalized word frequencies in question type T are p1, p2, p3...pN, then the frequency sum of the non-stop words in the current question in question type T is pH = p1+p2+p3+...+pN. Similarly, the frequency sum of the non-stop words in the current question in other question types is calculated. Finally, the sum of the frequency of the non-stop words in the current question in each question type is concatenated into a vector, and the dimension of the vector is equal to the total number of preset question types. The vector is normalized, and then feature extraction is performed to obtain the word frequency feature. Specifically, a three-layer fully connected network can be used as a word frequency feature extraction model. Other models can also be used in actual applications. The input and output dimensions of the model are the same. The embodiment of the present application can obtain a more detailed and accurate word frequency feature representation by calculating and normalizing the word frequency of the segmented words in each preset question type. This feature not only takes into account the occurrence of the segmented words in a single question type, but also eliminates the deviation between different segmented words through normalization processing, making the feature more stable and reliable. The word frequency feature can reflect the degree of correlation between the segmented words and different question types, which enables the model to more accurately identify the fine-grained type of the question in subsequent question type classification tasks, thereby improving recognition accuracy.

[0112] In an embodiment of the present application, optionally, the text feature extraction module performs stop word removal processing on the word segments obtained by word segmentation to obtain a word segmentation result, including: the text feature extraction module performs stop word removal processing on the word segments obtained by word segmentation based on the preset word statistics table to obtain a word segmentation result; or, the normalization processing of the word frequency of the word segments in each preset question type includes: calculating the total word frequency of the word segments in each preset question type, adding the total word frequency and a preset value to obtain a new total word frequency, and respectively calculating the ratio of the word frequency of the word segments in each preset question type to the new total word frequency as the normalized word frequency of the word segments in each preset question type.

[0113] In this embodiment, in order to avoid being unable to obtain relevant information of the word segments in the preset word statistics table, resulting in the inability to calculate question type weight features, word frequency features, etc. subsequently, the preset word statistics table can be used to perform stop word removal processing on the word segments, and all word segments that do not belong to the table are removed. If stop word removal processing is not performed according to the preset word statistics table, but only words with a very high appearance frequency but no substantial help in understanding the text content, such as "de", "le", "zai", etc., are removed. Then, for word segments that do not exist in the preset word statistics table, a preset value can be obtained as the number of occurrences of the word segments in each preset question type. In addition, in the word frequency normalization processing, after calculating the total word frequency of the word segments in each preset question type, the total word frequency and a preset value are added to obtain a new total word frequency, and then the ratio of the word frequency of the word segments in each preset question type to the new total word frequency is calculated. The advantage of this method is that by introducing a preset value, it is possible to prevent a calculation error caused by a denominator of 0 during calculation. And it can smooth the distribution of word frequencies to a certain extent and reduce the influence of extreme values on the normalization result. When the number of occurrences of a certain word segment in a certain question type is very small, according to the traditional normalization method, its normalized word frequency may be very close to 0, but after introducing the preset value, the normalized word frequency of this word segment in this question type will be relatively larger, thus retaining more information.

[0114] Step 203: Obtain the image features extracted by the image feature extraction module; calculate the image similarity between the image features and each image feature sample in the question image library, sort the image feature samples in descending order of image similarity, and select the first preset number of image feature samples ranked at the front to construct an image recognition result, where the image recognition result includes the first question type information and the image similarity corresponding to the first preset number of image feature samples.

[0115] In this embodiment, the image features extracted by the image feature extraction module can be directly obtained for subsequent image recognition. The similarity between the extracted image features and each image feature sample in the question image library is calculated, such as Euclidean distance, cosine similarity, etc. The image feature samples are further sorted in order from large to small according to the corresponding image similarity, and the image feature samples with a preset first number in the first sorting are taken as the candidate set. This preset first number is a threshold value, which is used to control the size of the candidate set for further processing and analysis in subsequent steps. Finally, an image recognition result is constructed based on the screened image feature samples. This result includes the first question type information and image similarity corresponding to the preset first number of image feature samples. The first question type information refers to the question type represented by these image feature samples, and the image similarity reflects the similarity between the extracted image features and these samples.

[0116] Step 204, calculate the text similarity between the question text and each question text sample in the question text library, sort the question text samples from large to small in text similarity, and take a preset second number of question text samples ranked first to construct a text recognition result, wherein the text recognition result includes the second question type information and text similarity corresponding to the preset second number of question text samples.

[0117] In this embodiment, similar to the above-mentioned calculation of the image recognition result, a text similarity calculation method (such as cosine similarity, Jaccard similarity, edit distance, etc.) is used to quantify the similarity between the question text and each question text sample in the question text library. According to the calculated text similarity, the question text samples in the question text library are sorted from large to small according to the text similarity, and a preset second number of question text samples ranked first are taken to construct the text recognition result. For each selected question text sample, the text recognition result should include the second question type information indicating the question type category to which the question text sample belongs and the similarity between the question text sample and the question text.

[0118] Step 205, traverse the image recognition results, and adjust the confidence of each preset question type corresponding to the first question type information of the currently traversed image feature sample according to the image similarity of the currently traversed image feature sample, so as to update the confidence of each preset question type.

[0119] In this embodiment, each image feature sample in the image recognition result is traversed. For each sample, the confidence of the preset question type is adjusted according to its image similarity. For example, if the first question type information of the currently traversed image feature sample is question type A, and the image similarity of the image feature sample is greater than a specific value, then the confidence of question type A in the multimodal recognition result is appropriately increased.

[0120] Step 206, traverse the text recognition results, and adjust the confidence of each preset question type corresponding to the second question type information of the currently traversed question text sample according to the text similarity of each traversed question text sample, so as to update the confidence of each preset question type.

[0121] In this embodiment, each text feature sample in the text recognition result is traversed. For each sample, the confidence of the preset question type is adjusted according to its text similarity. For example, if the second question type information of the currently traversed text feature sample is question type B, and the text similarity of the text feature sample is greater than a specific value, then the confidence of question type B in the multimodal recognition result is appropriately increased.

[0122] Step 207: Determine the question type of the question to be identified based on the updated confidence levels of the preset question types.

[0123] In this embodiment, the question type with the highest confidence is determined as the question type to be identified based on the confidence of each preset question type after update. If the confidence of multiple question types is close to the highest confidence, other strategies (such as considering the priority of the question type, historical data, etc.) can also be adopted to make the final judgment.

[0124] In addition, it should be noted that if two single-mode recognition methods, text and image, are used, Figure 2A If text-based unimodal recognition is used, the above steps 203 and 205 are not performed; if image-based unimodal recognition is used, the above steps 204 and 206 are not performed.

[0125] In the embodiment of the present application, optionally, the confidence corresponding to the first question type information of the image feature sample currently traversed in the confidence of each preset question type according to the image similarity of the image feature sample currently traversed in step 205 is adjusted, including: if the image similarity of the image feature sample currently traversed is greater than or equal to the preset image similarity, then based on the preset first increment value, the confidence corresponding to the first question type information of the image feature sample currently traversed in the confidence of each preset question type is increased and adjusted, otherwise the traversal of the image recognition result is terminated; In step 206, according to the text similarity of each traversed question text sample, the confidence of each preset question type corresponding to the second question type information of the currently traversed question text sample is adjusted to update the confidence of each preset question type, including: if the text similarity of the currently traversed question text sample is greater than or equal to the preset text similarity, then based on the preset second incremental value, the confidence of each preset question type corresponding to the second question type information of the currently traversed question text sample is increased and adjusted; otherwise, the traversal of the text recognition result is terminated.

[0126] In the above embodiment, in the process of adjusting the corresponding preset question type confidence according to the image similarity of each sample, first check whether the image similarity of the currently traversed image feature sample is greater than or equal to the preset image similarity. If the image similarity of the currently traversed image feature sample is greater than or equal to the preset image similarity, then increase the preset question type confidence corresponding to the first question type information of the sample according to the preset first increment value. This preset first increment value is a value set according to experience or experiment, and is used to control the amount of confidence increase. If the image similarity of the currently traversed image feature sample is less than the preset image similarity, it is considered that the sample is not similar enough to the image in the question to be identified, and there is no need to adjust the corresponding preset question type confidence. In addition, since the image similarity of each image feature sample is sorted from large to small, the subsequent image similarities will be less than the preset image similarity, so the traversal of the image recognition results can be terminated. The traversal process of the question text sample is similar to the above-mentioned traversal process of the image feature sample, which will not be repeated here. The embodiment of the present application adjusts the samples that are sufficiently similar to the question to be identified, which can ensure that the adjusted confidence level more accurately reflects the actual question type of the question to be identified, thereby improving the accuracy of recognition.

[0127] In the embodiment of the present application, optionally, before increasing and adjusting the confidence corresponding to the first question type information of the currently traversed image feature sample in the confidence of each preset question type based on the preset first increment value, the method further includes: if the image similarity of the currently traversed image feature sample is greater than or equal to the preset image similarity, then according to the target image similarity interval among the multiple preset image similarity intervals into which the image similarity of the currently traversed image feature sample falls, determining the preset first increment value corresponding to the target similarity interval, wherein the preset image similarity is the minimum value of the multiple preset image similarity intervals;

[0128] Before increasing and adjusting the confidence of each preset question type corresponding to the second question type information of the currently traversed question text sample based on the preset second incremental value, the method also includes: if the text similarity of the currently traversed question text sample is greater than or equal to the preset text similarity, then determining the preset second incremental value corresponding to the target text similarity interval according to the target text similarity interval among the multiple preset text similarity intervals into which the text similarity of the currently traversed question text sample falls, wherein the preset text similarity is the minimum value of the multiple preset text similarity intervals.

[0129] In the above embodiment, for the confidence adjustment of the image feature sample and the question text sample, not only whether the preset similarity threshold is met can be considered, but also the specific increment value can be determined according to the preset similarity interval into which the similarity falls. This refinement process can more accurately reflect the influence of different similarity levels on the confidence adjustment, thereby improving the accuracy and robustness of recognition. Specifically, when the image similarity of the currently traversed image feature sample is greater than or equal to the preset image similarity, it is determined which preset image similarity interval the similarity falls into. These preset image similarity intervals are set based on experience or experiments and are used to divide the similarity into different levels or ranges. According to the interval into which the similarity falls (i.e., the target image similarity interval), the preset first increment value corresponding to the interval is searched. This preset first increment value is set according to the similarity level of the interval and is used to control the amount of confidence increase. Then, this preset first increment value is used to increase the preset question type confidence corresponding to the first question type information of the currently traversed image feature sample. The refinement of the confidence adjustment for the question text sample is similar to the above process and will not be repeated here. Through the above-mentioned refinement processing, the specific increment value can be determined according to the interval into which the similarity falls, so as to more accurately adjust the confidence level and further improve the accuracy and robustness of the recognition.

[0130] Further, as a specific implementation of the method of FIG. 1 , the present application embodiment provides a question type recognition device based on a multimodal model, such as Figure 3 As shown, the device comprises:

[0131] A text recognition module is used to obtain a question image of a question to be recognized, and perform text recognition on the question image to determine the question text corresponding to the question image;

[0132] A multimodal recognition module, used to perform fusion recognition on the question image and the question text through a multimodal model to determine a multimodal recognition result of the question to be recognized;

[0133] A single-mode recognition module is used to match the question image according to the question image library to determine the image recognition result of the question to be recognized, and / or to match the question text according to the question text library to determine the text recognition result of the question to be recognized;

[0134] A question type determination module is used to determine the question type of the question to be identified based on the multimodal recognition result and the unimodal recognition result, wherein the unimodal recognition result is the text recognition result and / or the image recognition result.

[0135] In an embodiment of the present application, optionally, the multimodal model includes an image feature extraction module, a text feature extraction module, a word frequency feature extraction module, and a question type classification module; the multimodal recognition module is further used to:

[0136] Extracting image features corresponding to the title image by the image feature extraction module;

[0137] Perform word segmentation processing on the topic text by the text feature extraction module, and extract text features corresponding to the topic text based on the word segmentation result;

[0138] Extracting the word frequency feature corresponding to the title text according to the word segmentation result by the word frequency feature extraction module;

[0139] The question type classification module identifies the question type of the question to be identified based on the image features, the text features and the word frequency features to determine the confidence that the question to be identified belongs to each preset question type as the multimodal recognition result.

[0140] In the embodiment of the present application, optionally, the multimodal recognition module is further used to:

[0141] The text feature extraction module removes stop words from the segmented words obtained by the segmentation process to obtain a segmentation result;

[0142] For any word segmentation in the word segmentation result, extract the word feature of the word segmentation, obtain the number of occurrences of the word segmentation in each preset question type in the preset word statistical table, generate the question type weight feature corresponding to the word segmentation according to the number of occurrences of the word segmentation in each preset question type, and construct the question type weight feature and the word feature into the word segmentation feature of the word segmentation, wherein the preset word statistical table includes the number of occurrences of multiple preset words corresponding to each preset question type;

[0143] The text features corresponding to the title text are generated according to the segmentation features of each segmentation word.

[0144] In the embodiment of the present application, optionally, the device further includes: a statistical table construction module, which is used to:

[0145] Obtaining a sample set of questions, wherein the sample set of questions includes a plurality of sample questions corresponding to each preset question type;

[0146] Perform word segmentation and stop word removal on each question sample to obtain word segmentation information in each question sample;

[0147] Each segmentation word in the segmentation information is used as the preset word, the number of occurrences of the same preset word in each preset question type is counted, and the preset word statistics table is constructed.

[0148] In the embodiment of the present application, optionally, the multimodal recognition module is further used to:

[0149] For any segmentation in the segmentation results, according to the number of occurrences of the segmentation in each preset question type and the total number of occurrences of the segmentation in each preset question type, the word frequency of the segmentation in each preset question type is calculated respectively, and the word frequency of the segmentation in each preset question type is normalized;

[0150] The normalized word frequencies corresponding to each word segment belonging to the same preset question type are summed up to determine the summed word frequencies corresponding to each preset question type, and the word frequency features of the question text are constructed according to the summed word frequencies corresponding to each preset question type.

[0151] In the embodiment of the present application, optionally, the multimodal recognition module is further used to:

[0152] The text feature extraction module removes stop words from the segmented words obtained by the segmentation process based on the preset word statistics table to obtain a segmentation result; and / or,

[0153] Calculate the total word frequency of the segmented word in each preset question type, add the total word frequency to the preset value to obtain a new total word frequency, and calculate the ratio of the word frequency of the segmented word in each preset question type to the new total word frequency as the normalized word frequency of the segmented word in each preset question type.

[0154] In the embodiment of the present application, optionally, the single-mode recognition module is further used to:

[0155] Acquire the image features extracted by the image feature extraction module;

[0156] Calculate the image similarity between the image feature and each image feature sample in the question image library, sort the image feature samples from large to small in image similarity, and take a preset first number of image feature samples sorted first to construct the image recognition result, wherein the image recognition result includes the first question type information and image similarity corresponding to the preset first number of image feature samples.

[0157] In the embodiment of the present application, optionally, the single-mode recognition module is further used to:

[0158] Calculate the text similarity between the question text and each question text sample in the question text library, sort the question text samples from large to small in text similarity, and take a preset second number of question text samples ranked first to construct the text recognition result, wherein the text recognition result includes the second question type information and text similarity corresponding to the preset second number of question text samples.

[0159] In an embodiment of the present application, optionally, the multimodal recognition result includes the confidence that the question to be recognized belongs to each preset question type; the image recognition result includes the first question type information and image similarity corresponding to a preset first number of image feature samples; the text recognition result includes the second question type information and text similarity corresponding to a preset second number of question text samples;

[0160] The question type determination module is further used for:

[0161] Traversing the image recognition results, and adjusting the confidence corresponding to the first question type information of the currently traversed image feature sample in the confidence of each preset question type according to the image similarity of the currently traversed image feature sample, so as to update the confidence of each preset question type; and / or traversing the text recognition results, and adjusting the confidence corresponding to the second question type information of the currently traversed question text sample in the confidence of each preset question type according to the text similarity of each traversed question text sample, so as to update the confidence of each preset question type;

[0162] The question type of the question to be identified is determined according to the updated confidence levels of the preset question types.

[0163] In the embodiment of the present application, optionally, the question type determination module is further used to:

[0164] If the image similarity of the currently traversed image feature sample is greater than or equal to the preset image similarity, then based on the preset first increment value, the confidence of each preset question type corresponding to the first question type information of the currently traversed image feature sample is increased and adjusted, otherwise the traversal of the image recognition result is terminated;

[0165] If the text similarity of the currently traversed question text sample is greater than or equal to the preset text similarity, then based on the preset second incremental value, the confidence of each preset question type corresponding to the second question type information of the currently traversed question text sample is increased and adjusted, otherwise the traversal of the text recognition result is terminated.

[0166] In the embodiment of the present application, optionally, the question type determination module is further used to:

[0167] If the image similarity of the currently traversed image feature sample is greater than or equal to a preset image similarity, then according to a target image similarity interval among a plurality of preset image similarity intervals into which the image similarity of the currently traversed image feature sample falls, a preset first increment value corresponding to the target similarity interval is determined, wherein the preset image similarity is a minimum value of a plurality of preset image similarity intervals;

[0168] If the text similarity of the currently traversed question text sample is greater than or equal to the preset text similarity, then according to the target text similarity interval among the multiple preset text similarity intervals into which the text similarity of the currently traversed question text sample falls, a preset second incremental value corresponding to the target text similarity interval is determined, wherein the preset text similarity is the minimum value of the multiple preset text similarity intervals.

[0169] It should be noted that for other corresponding descriptions of the functional units involved in the question type recognition device based on a multimodal model provided in the embodiment of the present application, reference can be made to Figures 1A to 2B The corresponding description in the method will not be repeated here.

[0170] The embodiment of the present application also provides a computer device, which can be a personal computer, a server, a network device, etc. The computer device includes a bus, a processor, a memory and a communication interface, and can also include an input and output interface and a display device. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store location information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in each method embodiment are implemented.

[0171] Those skilled in the art will appreciate that the structure of the above-mentioned computer device is only a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components, or combine certain components, or have a different arrangement of components.

[0172] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and stores a computer program thereon. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0173] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0174] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0175] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0176] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A question type recognition method based on a multimodal model, characterized in that: The method comprises: Acquire a question image of a question to be identified, and perform text recognition on the question image to determine the question text corresponding to the question image; Performing fusion recognition on the question image and the question text through a multimodal model to determine a multimodal recognition result of the question to be recognized; Matching the question image according to the question image library to determine the image recognition result of the question to be recognized, and / or matching the question text according to the question text library to determine the text recognition result of the question to be recognized; The question type of the question to be recognized is determined according to the multimodal recognition result and the unimodal recognition result, wherein the unimodal recognition result is the text recognition result and / or the image recognition result.

2. The method according to claim 1, characterized in that The multimodal model includes an image feature extraction module, a text feature extraction module, a word frequency feature extraction module, and a question type classification module; The step of fusing and identifying the question image and the question text through a multimodal model to determine a multimodal recognition result of the question to be identified includes: Extracting image features corresponding to the title image by the image feature extraction module; Perform word segmentation processing on the topic text by the text feature extraction module, and extract text features corresponding to the topic text based on the word segmentation result; Extracting the word frequency feature corresponding to the title text according to the word segmentation result by the word frequency feature extraction module; The question type classification module identifies the question type of the question to be identified based on the image features, the text features and the word frequency features to determine the confidence that the question to be identified belongs to each preset question type as the multimodal recognition result.

3. The method according to claim 2, characterized in that The extracting of text features corresponding to the title text based on the word segmentation result includes: The text feature extraction module removes stop words from the segmented words obtained by the segmentation process to obtain a segmentation result; For any word segmentation in the word segmentation result, extract the word feature of the word segmentation, obtain the number of occurrences of the word segmentation in each preset question type in the preset word statistical table, generate the question type weight feature corresponding to the word segmentation according to the number of occurrences of the word segmentation in each preset question type, and construct the question type weight feature and the word feature into the word segmentation feature of the word segmentation, wherein the preset word statistical table includes the number of occurrences of multiple preset words corresponding to each preset question type; The text features corresponding to the title text are generated according to the segmentation features of each segmentation word.

4. The method according to claim 3, characterized in that The method for constructing the preset word statistical table includes: Obtaining a sample set of questions, wherein the sample set of questions includes a plurality of sample questions corresponding to each preset question type; Perform word segmentation and stop word removal on each question sample to obtain word segmentation information in each question sample; Each segmentation word in the segmentation information is used as the preset word, the number of occurrences of the same preset word in each preset question type is counted, and the preset word statistics table is constructed.

5. The method according to claim 3, characterized in that: The extracting the word frequency feature corresponding to the title text according to the word segmentation result by the word frequency feature extraction module includes: For any segmentation in the segmentation results, according to the number of occurrences of the segmentation in each preset question type and the total number of occurrences of the segmentation in each preset question type, the word frequency of the segmentation in each preset question type is calculated respectively, and the word frequency of the segmentation in each preset question type is normalized; The normalized word frequencies corresponding to each word segment belonging to the same preset question type are summed up to determine the summed word frequencies corresponding to each preset question type, and the word frequency features of the question text are constructed according to the summed word frequencies corresponding to each preset question type.

6. The method according to claim 5, characterized in that The text feature extraction module removes stop words from the segmented words obtained by the segmentation process to obtain the segmented words result, including: the text feature extraction module removes stop words from the segmented words obtained by the segmentation process based on the preset word statistics table to obtain the segmented words result; and / or, The normalizing process of the word frequency of the segmented word in each preset question type includes: calculating the total word frequency of the segmented word in each preset question type, adding the total word frequency to a preset value to obtain a new total word frequency, and respectively calculating the ratio of the word frequency of the segmented word in each preset question type to the new total word frequency as the normalized word frequency of the segmented word in each preset question type.

7. The method according to claim 2, characterized in that The step of matching the title image according to the title image library to determine the image recognition result of the title to be recognized includes: Acquire the image features extracted by the image feature extraction module; Calculate the image similarity between the image feature and each image feature sample in the question image library, sort the image feature samples from large to small in image similarity, and take a preset first number of image feature samples sorted first to construct the image recognition result, wherein the image recognition result includes the first question type information and image similarity corresponding to the preset first number of image feature samples.

8. The method according to claim 1, characterized in that The step of matching the question text according to the question text library to determine the text recognition result of the question to be recognized includes: Calculate the text similarity between the question text and each question text sample in the question text library, sort the question text samples from large to small in text similarity, and take a preset second number of question text samples ranked first to construct the text recognition result, wherein the text recognition result includes the second question type information and text similarity corresponding to the preset second number of question text samples.

9. The method according to any one of claims 1 to 8, characterized in that The multimodal recognition result includes the confidence that the question to be recognized belongs to each preset question type; the image recognition result includes the first question type information and image similarity corresponding to a preset first number of image feature samples; the text recognition result includes the second question type information and text similarity corresponding to a preset second number of question text samples; Determining the type of the question to be identified according to the multimodal recognition result and the unimodal recognition result includes: Traversing the image recognition results, and adjusting the confidence corresponding to the first question type information of the currently traversed image feature sample in the confidence of each preset question type according to the image similarity of the currently traversed image feature sample, so as to update the confidence of each preset question type; and / or traversing the text recognition results, and adjusting the confidence corresponding to the second question type information of the currently traversed question text sample in the confidence of each preset question type according to the text similarity of each traversed question text sample, so as to update the confidence of each preset question type; The question type of the question to be identified is determined according to the updated confidence levels of the preset question types.

10. The method according to claim 9, characterized in that The step of adjusting the confidence corresponding to the first question type information of the currently traversed image feature sample among the confidences of each preset question type according to the image similarity of the currently traversed image feature sample comprises: If the image similarity of the currently traversed image feature sample is greater than or equal to the preset image similarity, then based on the preset first increment value, the confidence of each preset question type corresponding to the first question type information of the currently traversed image feature sample is increased and adjusted, otherwise the traversal of the image recognition result is terminated; The step of adjusting the confidence of each preset question type corresponding to the second question type information of the currently traversed question text sample according to the text similarity of each traversed question text sample to update the confidence of each preset question type includes: If the text similarity of the currently traversed question text sample is greater than or equal to the preset text similarity, then based on the preset second incremental value, the confidence of each preset question type corresponding to the second question type information of the currently traversed question text sample is increased and adjusted, otherwise the traversal of the text recognition result is terminated.

11. The method according to claim 10, characterized in that Before increasing and adjusting the confidence of each preset question type corresponding to the first question type information of the currently traversed image feature sample based on the preset first increment value, the method further includes: If the image similarity of the currently traversed image feature sample is greater than or equal to a preset image similarity, then according to a target image similarity interval among a plurality of preset image similarity intervals into which the image similarity of the currently traversed image feature sample falls, a preset first increment value corresponding to the target similarity interval is determined, wherein the preset image similarity is a minimum value of a plurality of preset image similarity intervals; Before increasing and adjusting the confidence of each preset question type corresponding to the second question type information of the currently traversed question text sample based on the preset second increment value, the method further includes: If the text similarity of the currently traversed question text sample is greater than or equal to the preset text similarity, then according to the target text similarity interval among the multiple preset text similarity intervals into which the text similarity of the currently traversed question text sample falls, a preset second incremental value corresponding to the target text similarity interval is determined, wherein the preset text similarity is the minimum value of the multiple preset text similarity intervals.

12. A question type recognition device based on a multimodal model, characterized in that: The device comprises: A text recognition module is used to obtain a question image of a question to be recognized, and perform text recognition on the question image to determine the question text corresponding to the question image; A multimodal recognition module, used to perform fusion recognition on the question image and the question text through a multimodal model to determine a multimodal recognition result of the question to be recognized; A single-mode recognition module is used to match the question image according to the question image library to determine the image recognition result of the question to be recognized, and / or to match the question text according to the question text library to determine the text recognition result of the question to be recognized; A question type determination module is used to determine the question type of the question to be identified based on the multimodal recognition result and the unimodal recognition result, wherein the unimodal recognition result is the text recognition result and / or the image recognition result.

13. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

14. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 11 is implemented.

Images