Question type identification method, apparatus and device, and readable storage medium

Abstract

The invention provides a question type identification method, device and equipment and a readable storage medium, and the method comprises the steps: carrying out the feature extraction of a question image through a multi-mode large model, and representing the key features of a question type through a neuron combination activated in a question type classification full-connection network; the semantic, structure and context information of the question can be adaptively captured, misjudgment caused by language expression difference or format change is avoided, and the pre-constructed neuron combination library comprising the mapping relation between different question types and the corresponding activated neuron combination is utilized; and the target neuron combination is matched with the activated neuron combination in the question type classification full-connection network, so that efficient and accurate identification of question types of different languages and different cultural backgrounds can be realized based on the question type corresponding to the matched target neuron combination.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, apparatus, device, and readable storage medium for identifying a topic type. Background Art

[0002] The current homework grading system is mainly designed for a single language and fixed question types. It has limitations such as insufficient multi-language support, making it difficult to accurately parse questions in different languages, ineffectively identifying changes in question formats caused by differences in educational standards among countries, and a lack of understanding of cultural background, which makes it easy to misjudge questions containing specific cultural elements. As a result, the accuracy of question type recognition in different languages and question formats is low, such as the recognition of question types for overseas questions, which in turn affects the effectiveness of question grading. Summary of the Invention

[0003] In view of this, in order to solve the above technical problems, the present application provides a question type identification method, device, equipment and readable storage medium.

[0004] Specifically, this application is implemented through the following technical solutions:

[0005] According to a first aspect of an embodiment of the present application, a method for identifying question types is provided, the method comprising: in a process of identifying an input question image using a multimodal large model, obtaining a first neuron combination activated in a question type classification fully connected network in the multimodal large model; the first neuron combination comprising at least one neuron in the question type classification fully connected network;

[0006] Retrieving a target neuron combination that matches the first neuron combination from a pre-constructed neuron combination library; the neuron combination library at least includes the corresponding relationship between the question type identified by the training sample used in the training phase of the multimodal large model and the neuron combination activated in the question type classification fully connected network during the identification of the training sample; the matching degree between the question type identified by the training sample and the labeled type of the training sample meets the set requirements;

[0007] The question type corresponding to the question image is determined according to the target neuron combination and the question type corresponding to the target neuron combination.

[0008] Optionally, determining the topic type corresponding to the topic image according to the target neuron combination and the topic type corresponding to the target neuron combination includes:

[0009] If the target neuron combination is not marked with a set mark, determining the question type corresponding to the question image according to the question type corresponding to the target neuron combination; the set mark indicates that the target neuron combination does not meet the set classification requirements;

[0010] If the target neuron combination is marked with a set flag, then:

[0011] The confusion discrimination fully connected network in the multimodal large model is used to continue to identify the question image to obtain a second neuron combination activated in the confusion discrimination fully connected network, the first neuron combination and the second neuron combination are used to generate a third neuron combination, and a target neuron combination matching the third neuron combination is retrieved from the neuron combination library. If a target neuron combination is retrieved, the question type corresponding to the question image is determined based on the question type matched by the retrieved target neuron combination; if not retrieved, another model independent of the multimodal large model is used to identify the question type corresponding to the question image.

[0012] Optionally, the method further comprises:

[0013] In response to an event in which any neuron combination in the neuron combination library is marked with a set mark, the confusion discrimination fully connected network in the multimodal large model is used to continue to identify the question image corresponding to the neuron combination marked with the set mark, so as to obtain the neuron combination in which the confusion discrimination fully connected network in the multimodal large model is activated, and a new neuron combination is generated using the activated neuron combination and the neuron combination marked with the set mark, and the new neuron combination and its corresponding question type are stored in the neuron combination library.

[0014] Optionally, the multimodal large model also includes a feature extraction network; the output result of the feature extraction network is input into the question type classification fully connected network; or, when the confusion discrimination fully connected network in the multimodal large model is used to continue to identify the question image, the output result of the feature extraction network is input into the confusion discrimination fully connected network.

[0015] Optionally, the method further comprises a step of training the confusion discrimination fully connected network:

[0016] During the training phase of the multimodal large model, different sample features are input into the question type classification fully connected network when the same neuron combination in the question type classification fully connected network is activated;

[0017] The network parameters of the multimodal large model except the confusion discrimination fully connected network are kept unchanged, the different sample features are input into the confusion discrimination fully connected network, and the confusion discrimination fully connected network is used to predict whether the question types of the different sample features are the same to train the confusion discrimination fully connected network.

[0018] Optionally, the method further includes:

[0019] When no target neuron combination matching the first neuron combination is retrieved, using the question type classification fully connected network to output a predicted question type based on the first neuron combination as the question type of the question image;

[0020] The first neuron combination and the predicted question type are recorded in the neuron combination library in correspondence.

[0021] Optionally, the method further comprises: recognizing a question text based on the question image;

[0022] The question text and question image are used to perform text search and image search in the existing question library respectively, and when it is determined that the question represented by the question image does not belong to the questions in the existing question library, the input question image is recognized using a multimodal large model.

[0023] According to a second aspect of an embodiment of the present application, a question type identification device is provided, the device comprising:

[0024] a neuron combination acquisition module, configured to obtain a first neuron combination activated in a question type classification fully connected network in the multimodal large model during recognition of an input question image using the multimodal large model; the first neuron combination comprising at least one neuron in the question type classification fully connected network;

[0025] a neuron combination matching module, configured to retrieve a target neuron combination that matches the first neuron combination from a pre-constructed neuron combination library; the neuron combination library at least including the correspondence between the question type identified in the training sample used in the training phase of the multimodal large model and the neuron combinations activated in the question type classification fully connected network during the identification of the training sample; and the degree of matching between the question type identified in the training sample and the labeled type of the training sample meets a set requirement;

[0026] The question type determination module is used to determine the question type corresponding to the question image according to the target neuron combination and the question type corresponding to the target neuron combination.

[0027] Optionally, the question type determination module is specifically configured to:

[0028] If the target neuron combination is not marked with a set mark, determining the question type corresponding to the question image according to the question type corresponding to the target neuron combination; the set mark indicates that the target neuron combination does not meet the set classification requirements;

[0029] If the target neuron combination is marked with a set flag, then:

[0030] The confusion discrimination fully connected network in the multimodal large model is used to continue to identify the question image to obtain a second neuron combination activated in the confusion discrimination fully connected network, the first neuron combination and the second neuron combination are used to generate a third neuron combination, and a target neuron combination matching the third neuron combination is retrieved from the neuron combination library. If a target neuron combination is retrieved, the question type corresponding to the question image is determined based on the question type matched by the retrieved target neuron combination; if not retrieved, another model independent of the multimodal large model is used to identify the question type corresponding to the question image.

[0031] Optionally, the device also includes: in response to an event that any neuron combination in the neuron combination library is marked with a set mark, using the confusion discrimination fully connected network in the multimodal large model to continue to identify the question image corresponding to the neuron combination marked with the set mark, so as to obtain the neuron combination that is activated in the confusion discrimination fully connected network in the multimodal large model, using the activated neuron combination and the neuron combination marked with the set mark to generate a new neuron combination, and storing the new neuron combination and its corresponding question type in the neuron combination library.

[0032] Optionally, the multimodal large model also includes a feature extraction network; the output result of the feature extraction network is input into the question type classification fully connected network; or, when the confusion discrimination fully connected network in the multimodal large model is used to continue to identify the question image, the output result of the feature extraction network is input into the confusion discrimination fully connected network.

[0033] Optionally, the apparatus further comprises a step of training the confusion discrimination fully connected network:

[0034] During the training phase of the multimodal large model, different sample features are input into the question type classification fully connected network when the same neuron combination in the question type classification fully connected network is activated;

[0035] The network parameters of the multimodal large model except the confusion discrimination fully connected network are kept unchanged, the different sample features are input into the confusion discrimination fully connected network, and the confusion discrimination fully connected network is used to predict whether the question types of the different sample features are the same to train the confusion discrimination fully connected network.

[0036] Optionally, the device further comprises:

[0037] When no target neuron combination matching the first neuron combination is retrieved, using the question type classification fully connected network to output a predicted question type based on the first neuron combination as the question type of the question image;

[0038] The first neuron combination and the predicted question type are recorded in the neuron combination library in correspondence.

[0039] Optionally, the device further comprises:

[0040] Recognizing the title text based on the title image;

[0041] The question text and question image are used to perform text search and image search in the existing question library respectively, and when it is determined that the question represented by the question image does not belong to the questions in the existing question library, the input question image is recognized using a multimodal large model.

[0042] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory and a processor; the memory is used to store a computer program; and the processor is used to execute the above-mentioned question type identification method by calling the computer program.

[0043] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the above-mentioned question type identification method is implemented.

[0044] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0045] In the technical solution provided in the above-mentioned application, feature extraction of question images is performed through a multimodal large model, and the key features of the question type are represented by the combination of neurons activated in the fully connected network for question type classification. This can adaptively capture the semantics, structure and contextual information of the question, avoid misjudgment due to differences in language expression or format changes, and use a pre-constructed neuron combination library that includes mapping relationships between different question types and corresponding activated neuron combinations to match them with the activated neuron combinations in the fully connected network for question type classification. Based on the question type corresponding to the matched target neuron combination, efficient and accurate recognition of question types in different languages and cultural backgrounds can be achieved, thereby improving the international adaptability of the homework grading system and providing reliable technical support for intelligent assessment in transnational education scenarios.

[0046] It should be understood that the above general description and the detailed description below are merely exemplary and explanatory and cannot limit the present application. In addition, any embodiment in the present application does not necessarily achieve all the effects described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] Figure 1 This is a flowchart of a method for identifying question types shown in an exemplary embodiment of the present application;

[0049] Figure 2A This is a flow chart of a method for determining the question type corresponding to the question image based on the question type corresponding to the target neuron combination, as shown in an exemplary embodiment of the present application;

[0050] Figure 2B This is an example of a multimodal large model network structure shown in an exemplary embodiment of the present application;

[0051] Figure 3 This is a schematic diagram of a step of detecting whether a current question image to be identified belongs to a question in an existing question library, shown in an exemplary embodiment of the present application;

[0052] Figure 4 This is a structural diagram of a question type identification device shown in an exemplary embodiment of the present application;

[0053] Figure 5 It is a hardware schematic diagram of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0054] Here, exemplary embodiments will be described in detail, with examples shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Instead, they are merely examples of devices and methods consistent with certain aspects of this application as detailed in the appended claims. It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other.

[0055] Before introducing the topic type identification method provided by this application, some of the nouns involved in this application are briefly explained.

[0056] A deep neural network is a complex artificial neural network structure composed of multiple layers of neurons. Each neuron is responsible for receiving input, processing it, and producing output. A deep neural network can contain multiple hidden layers, which are located between the input and output layers.

[0057] Input layer: The first layer of the network, responsible for receiving input data, which is determined by the network position of the DNN. For example, if the DNN is a subnetwork within a task-specific network, the input layer receives the output of the previous layer of the task-specific network to which the DNN is connected.

[0058] Hidden layer: The core of the network. Each layer contains several neurons that process the data received from the previous layer and pass the results to the next layer. The number of hidden layers and the number of neurons in each layer can be set according to the complexity of the specific problem. In a narrow sense, the combination of hidden layer neurons can represent the activation state of the neurons in the previous network layer before the output layer (i.e., the layer of classification results) in the neural network. The activation state of these neurons (or called activation value, output value) will be passed to the output layer after certain calculations and processing, and the classification result will be obtained. Specifically, when the network receives an input sample, each neuron in the hidden layer will calculate its own activation value based on the input and weights. The different combinations of these activation values represent the feature representation of the input sample in the neural network. The feature representation is then passed to the output layer, and the output layer calculates the final classification result based on these feature representations and its own weights.

[0059] Output layer: The last or penultimate layer of the network. The number of neurons in the output layer depends on the requirements of the specific task (for example, the number of categories in a classification problem). In a neural network, this layer is responsible for calculating the final classification result based on the feature representation of the input sample. For classification tasks, the number of neurons in the output layer is typically equal to the number of categories, and the output value of each neuron represents the probability or score of the input sample belonging to the corresponding category.

[0060] Forward propagation refers to the process by which data in a neural network moves from the input layer through multiple hidden layers to the output layer. In each layer, the output of the previous layer becomes the input to the current layer. After a weighted summation, an activation function is applied to generate the output of the current layer. This process continues until the output layer is reached.

[0061] Backpropagation is the core algorithm used to train networks in deep learning. Its goal is to minimize the difference (i.e., error) between the network output and the actual label. This process is achieved through the following steps: the error between the predicted value and the true value is calculated at the output layer; the error is backpropagated to each layer of the network to calculate the error contribution of each neuron in each layer; and the weights in the neural network are adjusted using the gradient descent algorithm. Through multiple iterations of forward and backpropagation, deep neural networks gradually learn how to optimize task performance by adjusting their internal weights.

[0062] In recent years, with the acceleration of globalization and the increasing frequency of international educational exchanges, more and more students are choosing to study abroad or participate in international courses. These courses often include a variety of assignment types, such as multiple-choice questions, fill-in-the-blank questions, short-answer questions, and programming questions, and may involve different languages (such as English, French, and Spanish) and cultural backgrounds. To improve teaching efficiency, many institutions have begun to adopt automated homework grading systems to assist teachers in completing large-scale assignment assessments.

[0063] However, existing homework grading systems are primarily designed for question types in specific languages and countries. When adapting to overseas questions, they face limitations such as difficulty recognizing questions due to the diversity of question types, insufficient multilingual support, and cultural differences that affect semantic understanding. Specifically:

[0064] Education systems vary significantly across countries and regions, resulting in distinct question formats, presentation methods, and grading standards. For example, standardized tests in the United States (such as the SAT) differ significantly from the British A-Level exams in terms of question types and answer requirements. Existing grading systems, often based on fixed rules or template matching, struggle to adapt to the complexity and diversity of overseas questions, resulting in low accuracy in question type recognition.

[0065] Moreover, overseas questions are usually written in non-native languages (such as English, French, etc.), and the natural language processing capabilities of existing systems are limited, especially the ability to parse minority languages or specific subject terms (such as mathematical symbols, scientific formulas). For example, some systems may not be able to correctly distinguish between fill-in-the-blank questions (question à trous) in French and short-answer questions (short answer question) in English, thus affecting the accuracy of grading.

[0066] Furthermore, overseas questions may contain culturally specific knowledge or idiomatic expressions. For example, history questions may cover local events, while language questions may use slang or metaphors. Existing systems lack the ability to understand cultural context, making them prone to misjudgment due to semantic ambiguity.

[0067] In light of this, this application proposes a method for question type identification based on a multimodal large model and neuron combination analysis. This method analyzes the activated neuron combinations in the fully connected network for question type classification, combined with the neuron combination library established during the training phase, to accurately identify question types across languages and cultures. This improves the adaptability and automation level of the homework grading system for overseas questions, providing an efficient and reliable solution for intelligent homework grading in an international education environment.

[0068] The question type recognition method provided in this application is based on the neuron combination generated during the process of question type recognition of the input question image by a pre-trained multimodal large model. Among them, the multimodal large model described in this embodiment can extract feature information of different modalities of the input question image, such as text feature extraction and image feature extraction, and perform cross-modal fusion processing on the feature information of different modalities, and pass the fusion features obtained by the cross-modal fusion processing to the fully connected network layer, that is, Figure 1 The question type classification fully connected network shown is configured so that after receiving the fused features, the question type classification fully connected network transfers features between its internal network layers and activates corresponding neurons. The question type classification fully connected network supports using a combination of activated neurons to predict question types and outputs the predicted question type through the last layer of the fully connected network, i.e., the output layer. It will be understood that the question types described in this application represent different types of homework or exam questions that students need to complete in an educational environment, and may include but are not limited to multiple-choice questions, fill-in-the-blank questions, short-answer questions, programming questions, etc.

[0069] like Figure 1 A schematic diagram of the interactive steps of the question type identification method is shown as an example. The principle is that different neurons are activated in the question type classification fully connected network in the multimodal large model for different question types, and the feature extraction function of the multimodal large model is used to extract and fuse the feature information of different modalities of the input question image. The fused features after fusion processing are transferred to the question type classification fully connected network in the multimodal large model, and then the neuron combination (i.e., the first neuron combination) activated in the question type classification fully connected network after entering the question type classification fully connected network using the fused features is matched with the first neuron combination through the pre-stored correspondence between the question type and the neuron combination to identify the question type corresponding to the input question image.

[0070] Regarding the pre-training process of the multimodal large model, the training samples used in its training stage include question image samples with labeled question types, and feature information of different modalities is extracted and fused on the question image samples. The fused features after fusion processing are passed to the question type classification fully connected network in the multimodal large model to obtain the predicted question type generated by the output layer of the question type classification fully connected network, and the loss function is calculated based on the predicted question type and the labeled question type of the question image sample. The network parameters of each network layer of the multimodal large model are updated using the backpropagation algorithm until the predetermined training convergence conditions are reached, thereby obtaining a trained multimodal large model. The multimodal large model uses the feature processing capabilities learned during training to reflect the feature patterns corresponding to different question types through different neuron activation states in the question type classification fully connected network through layer-by-layer network processing in the inference stage.

[0071] Based on the above pre-trained multimodal large model, such as Figure 1 As shown, the topic type identification method provided in this application can be implemented by at least the following steps:

[0072] S101, in the process of using the multimodal large model to recognize an input question image, obtaining a first neuron combination activated in a question type classification fully connected network in the multimodal large model; the first neuron combination includes at least one neuron in the question type classification fully connected network;

[0073] A fully connected network for question type classification consists of at least one hidden layer and an output layer. The hidden layer consists of at least one network layer. Each network layer contains several neurons that process data received from the previous layer and pass the results to the next layer. The number of hidden layers and the number of neurons in each layer can be set based on the complexity of the specific problem. In the classification task of question type identification, the number of neurons in the output layer is equal to the number of categories corresponding to the preset question types, and each neuron corresponds to the predicted probability of a question type category.

[0074] The activation state of the hidden layer neurons is determined by the input data and the weight and bias value of the neuron. This can be achieved by applying an activation function, which is used to map the weighted input of the neuron (that is, the sum of the product of the input data and the weight plus the bias value) to an output value, and then determine whether the neuron is activated by detecting whether the output value is greater than a set threshold. The set threshold can be flexibly set according to the properties of the activation function and the actual task requirements.

[0075] In this embodiment, the meaning of a neuron being activated is that the output value of the data received by the neuron after being processed by the activation function is greater than a set threshold, indicating that the neuron is activated; otherwise, it indicates that the neuron is not activated. The set threshold value can be set to 0.5, and in actual application, the value of the threshold value can also be adjusted according to the application scenario. It should be clarified that the neuron combination described in this application represents a combination formed by activated neuron types, reflecting the neural network's understanding and response to the characteristics of the question type of a specific question. For example, if the output values corresponding to neurons 2, 3, 5, and 6 are greater than the set threshold, then the first neuron combination represents the combination formed by neurons 2, 3, 5, and 6.

[0076] In the present embodiment, during the process of using the multimodal large model to recognize the input question image, when the input question image is subjected to feature processing by the multimodal large model to obtain a cross-modal fusion feature and the cross-modal fusion feature is input into the question type classification fully connected network of the multimodal large model, the question type prediction function based on the activated neuron combination of the question type classification fully connected network itself is not used to directly output the predicted question type corresponding to the question image through the output layer. Instead, the first neuron combination activated in the question type classification fully connected network after the cross-modal fusion feature is input into the question type classification fully connected network is first obtained.

[0077] Among them, the first neuron combination comes from the hidden layer in the question type classification fully connected network. The previous hidden layer based on the output layer can provide a highly abstract and high-level feature representation, and the feature representation is close enough to the final classification task and can well reflect the intrinsic structure and pattern of the input data. Therefore, the combination formed by the neurons activated in the previous hidden layer of the output layer of the question type classification fully connected network (that is, the penultimate layer of the question type classification fully connected network) can be used as the first neuron combination.

[0078] For example, taking the first activated neuron combination in the penultimate layer of the fully connected network for question type classification (i.e., the hidden layer connected to the output layer) as an example, assuming that the penultimate layer includes neurons 1-6, the data input to the penultimate layer is processed by the activation function to obtain the output values of each neuron as [0, 0.67, 0.88, 0.35, 0.91, 0.53] in sequence, and the threshold is set to 0.5 to determine whether the neuron is activated, then the first neuron combination can reflect the activated neurons 2, 3, 5, and 6. For example, the first neuron combination can be expressed as a combination of neurons 2, 3, 5, and 6 [2, 3, 5, 6], or it can also be expressed as [2: 0.67, 3: 0.88, 5: 0.01, 6: 0.53].

[0079] Regarding the method of obtaining the first activated neuron combination in the fully connected network of question type classification, it can be achieved by viewing and modifying the configuration file of the fully connected network of question type classification so that it outputs the activated neuron combination in the specified hidden layer during the forward reasoning process; or, it can be achieved by using the API provided by the neural network framework to obtain the activated neuron combination in the specified hidden layer, such as using the API provided by the framework to register a custom hook function to the output of the specified hidden layer, and automatically calling the hook function when the forward propagation passes through the hidden layer of the registered hook function to capture the output of the specified hidden layer, wherein the hook function will receive the output of the specified hidden layer as input, and can perform any required operations (such as printing output, executing algorithmic operations based on the input). It can be understood that the above examples are only some exemplary implementation methods of obtaining activated neuron combinations from a neural network. Those skilled in the art can also use other methods in the relevant technology that can obtain activated neurons to obtain the first activated neuron combination in the fully connected network of question type classification, and this application does not limit this.

[0080] S102: Retrieving a target neuron combination that matches the first neuron combination from a pre-constructed neuron combination library; the neuron combination library at least includes the correspondence between the question type identified in the training sample used in the training phase of the multimodal large model and the neuron combination activated in the question type classification fully connected network during the identification of the training sample; the matching degree between the question type identified in the training sample and the labeled type of the training sample meets the set requirements;

[0081] The neuron combination library represents a data structure that stores multiple question types and at least one neuron combination corresponding to each question type. Question images of the same question type may activate different neuron combinations in the fully connected network for question type classification in a multimodal large model. Therefore, the neuron combination library stores multiple question types, each corresponding to one or more neuron combinations.

[0082] The neuron combinations corresponding to the question types stored in the neuron combination library represent the neuron combinations activated within the question type classification fully connected network of the multimodal large model when a question image belonging to that question type was correctly predicted by the question type classification fully connected network. Correct prediction by the question type classification fully connected network of the multimodal large model can be understood as the degree of match between the predicted type output by the output layer of the question type classification fully connected network in the multimodal large model and the label type of the question image meeting the set requirements. The label type represents the label or category assigned during manual or automatic labeling of the training sample.

[0083] In this embodiment, the neuron combination library can be constructed based on training samples during the training phase of the multimodal large model. For training samples whose predicted type output by the output layer of the question type classification fully connected network of the multimodal large model during the training phase meets the set requirements for matching the labeled type, such as training samples whose textual / semantic matching of the type is greater than a set matching threshold, the corresponding relationship between the labeled type / predicted type of the training sample and the neuron combination activated by the training sample in the question type classification fully connected network can be stored in the neuron combination library.

[0084] For example, the annotation type of the question image sample T1 is "multiple choice question". After being feature processed by the multimodal large model and passed to the question type classification fully connected network, the neuron combination M1 is activated in the penultimate layer of the question type classification fully connected network. Based on the activated neuron combination in the penultimate layer, the prediction result output by the output layer of the question type classification fully connected network is "multiple choice question", which meets the requirements of the annotation type matching degree. Then the corresponding relationship "multiple choice question": "M1" is stored in the neuron combination library.

[0085] The correspondence between the question types and neuron combinations contained in the neuron combination library is dynamically updated during the training or forward reasoning stage, such as adding new neuron combinations and their corresponding question types that are not stored in the neural cloud combination library, or, in the case where the question type is incorrectly identified based on the question type corresponding to the stored neuron combination, updating the stored neuron combination, including but not limited to setting a set mark for the stored neuron combination and adding a new remedial method for question type identification based on the stored neuron combination.

[0086] For the input question image, after obtaining the first neuron combination activated in the question type classification fully connected network of the multimodal large model, a search is performed from the stored neuron combinations in the neuron combination library to retrieve the same stored neuron combination as the neurons included in the first neuron combination. For example, the penultimate layer of the question type classification fully connected network of the multimodal large model contains 12 neurons. The relationship between the question types and neuron combinations stored in the neuron combination library is as follows:

[0087] Multiple Choice Questions [2、3、5、6]，[2、4、5、6]，…… True or False [1、4、7、9、10]，…… Fill in the blanks …… …… ……

[0088] If the first neuron combination obtained is m = [2, 3, 5, 6], then the stored neuron combinations are searched to see whether there is a neuron combination of [2, 3, 5, 6]. If a completely matching stored neuron combination is retrieved, then the stored neuron combination is used as the target neuron combination that matches the first neuron combination.

[0089] S103 , determining the question type corresponding to the question image according to the target neuron combination and the question type corresponding to the target neuron combination.

[0090] In this embodiment, when a target neuron combination matching the first neuron combination is retrieved from the aforementioned neuron combination library, the question type corresponding to the input question image can be determined based on whether the target neuron combination carries a set mark and the question type corresponding to the target neuron combination. The set mark is used to indicate that the target neuron combination does not meet the set classification requirements. The set mark can be added to the neuron combination that incorrectly identified the question type by providing feedback information indicating an error in the question type identified based on the neuron combination. The presence of the set mark in the target neuron combination indicates that an error in question type identification occurred during the process of identifying the question type based on the target neuron combination, and that confusion exists in the question type identification and distinction of the target neuron combination.

[0091] When the target neuron combination is retrieved, if the target neuron combination is not marked with a set mark, the question type corresponding to the target neuron combination in the neuron combination library can be obtained, and the question type corresponding to the target neuron combination can be used as the question type corresponding to the question image. If the target neuron combination is marked with a set mark, the question image is further recognized using another confusion discrimination fully connected network in the multimodal large model that is a parallel subnetwork to the question type classification fully connected network. Based on the second neuron combination activated in the confusion discrimination fully connected network and the first neuron combination activated in the question type classification fully connected network, the question type corresponding to the question image is identified.

[0092] When the target neuron combination matching the first neuron combination is not retrieved, that is, the first neuron combination activated by the currently input question image in the question type classification fully connected network belongs to a new neuron combination outside the neuron combination library, see Figure 1 In step S104 shown, the question type prediction capability learned by the question type classification fully connected network during the training phase can be utilized. Based on the first neuron combination, the predicted question type can be directly outputted as the question type of the question image through the output layer of the question type classification fully connected network. Based on this, since the target neuron combination matching the first neuron combination has not been retrieved, it indicates that the first neuron combination currently activated in the question type classification fully connected network is a new neuron combination that has not been recorded in the neuron combination library. Therefore, after obtaining the predicted question type output by the question type classification fully connected network, the first neuron combination and the predicted question type can be recorded in the neuron combination library in correspondence, thereby achieving dynamic updating of the neuron combination library, enriching and improving the content of the neuron combination library, and improving the classification efficiency and accuracy of subsequent similar question images.

[0093] In the disclosed embodiment, feature extraction is performed on question images using a multimodal large model, and the key features of the question type are represented by the activated neuron combination in the question type classification fully connected network. This method can adaptively capture the semantic, structural, and contextual information of the question, avoiding misjudgment due to differences in language expression or format changes. A pre-constructed neuron combination library, which includes a mapping relationship between different question types and corresponding activated neuron combinations, is used to match them with the activated neuron combination in the question type classification fully connected network. The matched target neuron combination corresponds to the question type. The neuron combination library can support joint learning of English, French, and Spanish questions in multilingual training samples. The matching mechanism of the neuron combination library can realize the recognition of question types in different languages and cultural backgrounds, thereby achieving efficient and accurate recognition of various questions in complex question formats or language expressions, such as overseas question types. This reduces the limitations of question type recognition in related technologies caused by language barriers, cultural differences, and question diversity, improves the international adaptability of homework grading systems, and provides reliable technical support for intelligent assessment in transnational education scenarios.

[0094] In some embodiments, due to the problem of question type identification errors when identifying question types based on neuron combinations, for the aforementioned embodiments, the question type corresponding to the question image is determined based on the target neuron combination and the question type corresponding to the target neuron combination. After retrieving the target neuron combination that matches the first neuron combination from the neuron combination library, it is necessary to adopt different question type determination methods based on whether the target neuron combination belongs to a combination that causes question type identification confusion or errors. Whether the target neuron combination belongs to a combination that causes question type identification confusion or errors can be determined by detecting whether the target neuron combination is set or marked with a set flag.

[0095] This set flag is added to the target neuron combination in response to feedback information from a previous historical reasoning process indicating that the question type determined based on a matching target neuron combination retrieved from the neuron combination library does not match the actual question type. This flag is used to indicate that this neuron combination may have confused or incorrectly identified the question type. For example, during a historical forward reasoning process, a target neuron combination M2 that matches the first neuron combination of the question image to be identified is retrieved from the neuron combination library. The question type corresponding to M2 is used as the question type of the question image to be identified. If feedback is received indicating that the question type of the question image to be identified was incorrect, a set flag is added to the target neuron combination M2, indicating that confusion exists in the question type identified based on the target neuron combination.

[0096] In the case where the feedback information triggers the addition of a set mark to the neuron combination library, in response to the event that any neuron combination in the neuron combination library is marked with a set mark, the confusion discrimination fully connected network in the multimodal large model can be used to continue to identify the question image corresponding to the neuron combination marked with the set mark (that is, the question image with incorrect question type recognition recorded in the feedback information), that is, the feature information of the question image processed by the multimodal large model is input into the question type classification fully connected network, and then input into the confusion discrimination fully connected network in the multimodal large model to obtain the activated neuron combination of the confusion discrimination fully connected network in the multimodal large model, and a new neuron combination is generated using the activated neuron combination and the neuron combination marked with the set mark, and the new neuron combination is stored in the neuron combination library in correspondence with the true / correct question type of the question image recorded in the feedback information.

[0097] The confusion discrimination fully connected network and the question type classification fully connected network are two parallel subnetworks within the multimodal model. The confusion discrimination fully connected network can be trained by predicting whether the question types of specific sample pairs are the same. Its prediction results reflect whether the question types of the input sample pairs are the same. The specific sample pairs can at least include question sample pairs with the same combination of activated neurons in the question type classification fully connected network.

[0098] For example, still taking the target neuron combination M2 matched from the neuron combination library to the first neuron combination activated in the question type classification fully connected network in the aforementioned historical forward reasoning process, and receiving feedback information indicating that the question type of the question image to be identified is incorrectly identified, in response to adding a set mark to the target neuron combination M2, the question image to be identified is triggered to be subjected to the multimodal large model for feature information extraction of different modalities and cross-modal fusion, and then input into the feature information of the question type classification fully connected network, and input into the confusion discrimination fully connected network in the multimodal large model, thereby obtaining the activated neuron combination M2' in the confusion discrimination fully connected network, and using [M2+M2'] as a new neuron combination, and the correct question type of the question image to be identified recorded in the feedback information, such as "true or false question", and the new neuron combination [M2+M2'] are updated and stored in the neuron combination library accordingly.

[0099] Regarding adding a setting tag, the setting tag can be used as an additional attribute or associated value of the neuron combination, and the setting tag and the neuron combination with the setting tag are additionally stored in the neuron combination library. For example, the neuron combination library also includes storage records of neuron combinations marked with setting tags, such as setting tags: {[1, 3, 4, 5, 6], [2, 4, 7, 8, 9], ...}; or, the setting tag can be directly added to the neuron combination as part of the neuron combination content, for example, the setting tag value is different from the values of all neuron types, such as "X". If the neuron combination M2 = [1, 3, 4, 5, 6] is to be added with a setting tag, the value "X" can be directly added to the end of the combination content of M2 to obtain M2 = [1, 3, 4, 5, 6; X] after the tag is added.

[0100] Based on this, see Figure 2A A flowchart of a method for determining the question type corresponding to a question image based on the question type corresponding to a target neuron combination is shown as an example. In the current reasoning process, when a target neuron combination matching the first neuron combination is retrieved, the question type corresponding to the question image can be determined in the following manner:

[0101] S201, determining whether the target neuron combination is marked with a set mark; the set mark indicates that the target neuron combination does not meet the set classification requirements;

[0102] S202, if the target neuron combination is not marked with a set mark, determining the question type corresponding to the question image according to the question type corresponding to the target neuron combination;

[0103] If the target neuron combination is not marked with a set flag, it means that the determination of the question type based on the target neuron combination in the current context is considered reliable. Therefore, the question type corresponding to the target neuron combination can be directly used as the question type corresponding to the question image.

[0104] S203: If the target neuron combination is marked with a set flag, the confusion discrimination fully connected network in the multimodal large model is used to continue recognizing the question image, including:

[0105] S2031, continue to recognize the question image using the confusion discrimination fully connected network in the multimodal large model to obtain a second neuron combination activated in the confusion discrimination fully connected network;

[0106] Among them, see Figure 2BAn example of a multimodal large model network structure is shown as an example. The confusion discrimination fully connected network and the question type classification fully connected network are two parallel fully connected networks in the multimodal large model. In the process of using the multimodal large model to identify the question type of the input question image, the question type classification fully connected network is first used to obtain the first neuron combination. When the target neuron combination matched based on the first neuron combination in the question type classification fully connected network is marked with a set mark, the question type corresponding to the target neuron combination cannot be directly used to determine the question type of the question image, and then the confusion discrimination fully connected network is used to continue to identify the question image. During the identification process, the feature information input to the confusion discrimination fully connected network is the feature information input to the question type classification fully connected network.

[0107] Based on this multimodal large model, it has the ability to extract feature information of different modalities and perform cross-modal feature information fusion, such as Figure 2B As shown, the multimodal large model may also include a feature extraction network, which is used to extract features of the input question image from different modalities and fuse the feature information of different modalities. The output result of the feature extraction network is input into the question type classification fully connected network; or, when the confusion discrimination fully connected network in the multimodal large model is used to continue to identify the question image, the output result of the feature extraction network is input into the confusion discrimination fully connected network.

[0108] The feature extraction network may include at least a text feature extraction network, an image feature extraction network and a cross-modal feature fusion network. The text feature extraction network is used to extract features from the text carried in the input question image, the image feature vector extraction network is used to extract features from the question image, and the cross-modal feature fusion network is used to perform cross-modal fusion processing on the output results from the text feature extraction network and the image feature extraction network.

[0109] The meaning of the second neuron combination is similar to that of the first neuron combination, and reference may be made to the explanation of the first neuron combination. The second neuron combination originates from the hidden layer in the confusion discriminant fully connected network. The previous hidden layer based on the output layer can provide a highly abstract and high-level feature representation, and this feature representation is close enough to the final classification task and can well reflect the intrinsic structure and pattern of the input data. Therefore, the combination formed by the neurons activated in the previous hidden layer of the output layer of the confusion discriminant fully connected network (i.e., the penultimate layer of the confusion discriminant fully connected network) can be used as the second neuron combination.

[0110] S2032, generating a third neuron combination by using the first neuron combination and the second neuron combination;

[0111] Because the second neuron combination comes from the confusion discriminant fully connected network, the purpose of obtaining the second neuron combination in this embodiment is to enhance the feature representation of the question image. Therefore, the first neuron combination and the second neuron combination can be directly spliced to form a new neuron combination containing the content of the two neuron combinations, namely the third neuron combination. For example, if the first neuron combination is [2, 3, 4, 5] and the second neuron combination is [1, 4, 6, 7], the third neuron combination can be represented as [2, 3, 4, 5; 1, 4, 6, 7].

[0112] S2033: Retrieve a target neuron combination matching the third neuron combination from the neuron combination library, and determine the question type corresponding to the question image based on whether a target neuron combination matching the third neuron combination is retrieved.

[0113] That is, as can be seen from the aforementioned embodiments, if an event is triggered that marks any neuron combination in the neuron combination library, the corresponding relationship between the new neuron combination, including the dual neuron combination, and its corresponding question type will be dynamically updated in the neuron combination library. Therefore, during the current reasoning process, for the third neuron combination obtained by the question type classification fully connected network and the confusion discrimination fully connected network for the currently input question image, the target neuron combination can be first searched from the neuron combination library to see if it matches the third neuron combination:

[0114] S2033a, if a target neuron combination matching the third neuron combination is retrieved, the question type corresponding to the question image can be determined based on the question type matching the retrieved target neuron combination, that is, the question type matching the retrieved target neuron combination can be directly used as the question type corresponding to the question image.

[0115] S2033b: If the target neuron combination matching the third neuron combination is not retrieved, another pre-trained model independent of the multimodal large model can be used to identify the topic type corresponding to the topic image.

[0116] That is, the question image input into the multimodal large model can be directly input into the other model, so that the other model can identify the question type of the question image based on the feature processing and prediction capabilities learned in its training phase, thereby obtaining the question type prediction result output by the other model as the question type of the question image.

[0117] For example, taking the aforementioned third neuron combination [2, 3, 4, 5; 1, 4, 6, 7] as an example, if a stored neuron combination such as M6 = [2, 3, 4, 5; 1, 4, 6, 7] is also retrieved from the neuron combination library, the question type corresponding to M6 in the neuron combination library is directly used as the question type for the question image. Conversely, if M6 is not retrieved from the neuron combination library, another model independent of the multimodal large model is used to predict the question type for the question image.

[0118] Among them, the training steps of the other model may include: obtaining a first question sample in which the first neuron combination activated in the question type classification fully connected network belongs to a neuron combination marked with a set mark in the neuron combination library, as a training sample, and the first question sample is marked with a question type; inputting the question image of the first question sample into the other model independent of the multimodal large model to obtain the predicted question type generated by the other model; calculating the loss of the predicted question type and the marked question type of the first question sample, and adjusting the network parameters of the other model by minimizing the loss until the predetermined training convergence condition is reached.

[0119] In the embodiment disclosed herein, by introducing a collaborative processing method of a neuron combination discrimination mechanism based on set labels and a confusion discrimination fully connected network, the recognition process of the confusion discrimination fully connected network is triggered based on the pre-labeled easily confused neuron combination, and the first neuron combination in the question type classification fully connected network is combined with the second neuron combination in the confusion discrimination fully connected network to generate a more discriminative third neuron combination for matching, thereby enhancing the feature expression of the question image to be identified, avoiding the limitations of a single neuron combination matching when the boundary of the question type feature is fuzzy, and thus improving the accuracy of question type recognition. In addition, for double matching failures caused by training data deviations or rare question types, that is, the third neuron combination also fails to match, another trained model independent of the multimodal large model is automatically enabled for supplementary prediction, thereby effectively dealing with abnormal recognition problems of different question types, enhancing the fault tolerance processing capability for abnormal recognition of question types, and improving the overall accuracy of question type recognition.

[0120] In some embodiments, for the confusion discrimination fully connected network described in the aforementioned embodiments, the confusion discrimination fully connected network is designed to process question images that cannot be accurately recognized by the question type classification fully connected network, that is, the confusion discrimination fully connected network is used to supplement or correct the errors or uncertainties of the question type classification fully connected network. For the sake of task specificity and to avoid overfitting of the multimodal large model, when training the confusion discrimination fully connected network, the network parameters other than the confusion discrimination fully connected network in the pre-trained multimodal large model are frozen, and the network parameters of the confusion discrimination fully connected network are adjusted separately. The training can be completed in the following manner:

[0121] During the training phase of the multimodal large model, different sample features are input into the question type classification fully connected network when the same neuron combination in the question type classification fully connected network is activated; network parameters other than the confusion discrimination fully connected network in the multimodal large model are kept unchanged, the different sample features are input into the confusion discrimination fully connected network, and the confusion discrimination fully connected network is used to predict whether the question types of the different sample features are the same to train the confusion discrimination fully connected network.

[0122] For example, if there are samples A, B, C, and D that have the same combination of neurons activated in the question type classification fully connected network, then the sample features of at least two samples (such as A and B, A and CD) that are input into the question type classification fully connected network can be input into the confusion discrimination fully connected network together, so that the confusion discrimination fully connected network processes the at least two sample features input separately, and outputs a prediction result indicating whether the labeled question types corresponding to the two sample features are the same, for example, a prediction result of 0 indicates different, and a prediction result of 1 indicates the same, and then adjusts the network parameters of the confusion discrimination fully connected network by calculating the loss, so that the confusion discrimination fully connected network can learn to maximize the distinction between samples that have the same combination of neurons activated in the question type classification fully connected network but belong to different question types. By continuously calculating and adjusting the loss, the confusion discrimination fully connected network can gradually optimize its parameters to more accurately identify and process question images that are difficult for the question type classification fully connected network to distinguish, thereby improving the accuracy and robustness of the entire model.

[0123] In the disclosed embodiment, by freezing the network parameters of the pre-trained multimodal large model except for the confusion discrimination fully connected network, a targeted training method is adopted in which the network parameters of the confusion discrimination fully connected network are adjusted separately. This enables the confusion discrimination network to solve specific problems more effectively, significantly reduces training time and computing resource consumption, avoids unnecessary adjustments to the already stable and well-performing network parts in the multimodal large model, reduces the risk of overfitting, and maintains the stability of the entire multimodal large model.

[0124] In some embodiments, in order to improve the overall efficiency of question type identification and optimize resource utilization, this embodiment provides a hierarchical identification method. When it is determined that the question represented by the current question image to be identified does not belong to the existing questions in the question database, the multimodal large model provided by this application is used to identify the input question image. If the question represented by the current question image to be identified belongs to the existing questions in the question database, the question type can be directly determined based on the existing question in the question database. Based on this, before the aforementioned step S101, the method can also include the following process of detecting whether the current question image to be identified belongs to the questions in the existing question database, namely:

[0125] The question text is identified based on the question image; the question text and the question image are used to perform text search and image search in an existing question library respectively; and when it is determined that the question represented by the question image does not belong to the questions in the existing question library, the input question image is identified using a multimodal large model.

[0126] See also Figure 3 The schematic diagram of the steps for detecting whether the current question image to be identified belongs to a question in an existing question library is shown as an example. The question text can be identified from the question image through text recognition technology such as OCR (Optical Character Recognition) technology, and text search S301 and image search S302 can be performed simultaneously. The existing question library can include a text search library and an image search library. The same question can be associated through a question identifier, and the corresponding question text and question image are stored in the text search library and the image search library respectively:

[0127] The text search step S301 may include:

[0128] S3011, retrieving the first question with the highest text similarity to the question text from the text search library;

[0129] S3012, determining whether the text similarity is greater than a first threshold and the text length of the question text is greater than a text threshold;

[0130] S3013a, if yes, then determine the title type of the first title as the title type as the title image;

[0131] S3013b, if not, determining that the title image does not belong to the text search library;

[0132] The image search step S302 includes:

[0133] S3021, retrieving a second question having the highest image similarity to the question image from an image search library;

[0134] S3022, determining whether the image similarity is greater than a second threshold;

[0135] S3023a: If yes, determine the question type of the second question as the question type of the question image;

[0136] S3023b, if not, determining that the title image does not belong to the image search library;

[0137] S303: When the question image does not belong to the text search library and the image search library, that is, the question represented by the question image does not belong to the questions in the existing question library, the input question image is recognized using the multimodal large model.

[0138] In the disclosed embodiment, by first performing a text search and an image search on the question image, it is possible to quickly determine whether the question belongs to an existing question library. This method combines text search and image search, and can handle question images of different qualities and formats. If the question belongs to an existing question library, the question type information in the library can be directly used, avoiding unnecessary startup of a large multimodal model for question type identification, reducing the consumption of computing resources, shortening the waiting time for question type identification, and improving the response speed.

[0139] In some embodiments, the neuron combinations stored in a pre-built neuron combination library can be updated in the following manner: in response to receiving feedback information indicating a question classification error, if a matching neuron combination exists in the neuron combination library that matches the fourth neuron combination of the incorrectly classified question, and the fourth neuron combination represents the neuron combination activated in the question type classification fully connected network for the incorrectly classified question, then: the matching neuron combination is marked with a flag; in response to the event that the matching neuron combination is marked with a flag, a fifth neuron combination activated in the confusion discrimination fully connected network for the incorrectly classified question is obtained; the fourth neuron combination and the fifth neuron combination are concatenated as a new neuron combination, and the new neuron combination and the actual question type of the incorrectly classified question indicated by the feedback information are stored in the neuron combination library accordingly. If a matching neuron combination does not exist in the neuron combination library that matches the fourth neuron combination of the incorrectly classified question, the fourth neuron combination and the actual question type of the incorrectly classified question indicated by the feedback information are directly stored in the neuron combination library accordingly.

[0140] In the embodiment of the present disclosure, by allowing the neuron combination library to be quickly updated when feedback information of incorrect question classification is received, the question type of the question image that was incorrectly identified online can be corrected immediately, avoiding the tedious and time-consuming process of retraining the entire model, allowing the system to quickly adapt and correct errors, thereby improving the accuracy and efficiency of online recognition.

[0141] Corresponding to the embodiment of the above-mentioned method for identifying the topic type, see Figure 4 As shown, the present application also provides an embodiment of a device for identifying a topic type, the device comprising:

[0142] A neuron combination acquisition module 401 is configured to obtain a first neuron combination activated in a question type classification fully connected network in the multimodal large model during recognition of an input question image using the multimodal large model; the first neuron combination includes at least one neuron in the question type classification fully connected network;

[0143] A neuron combination matching module 402 is configured to retrieve a target neuron combination that matches the first neuron combination from a pre-built neuron combination library; the neuron combination library at least includes the corresponding relationships between the question types identified in the training samples used in the training phase of the multimodal large model and the neuron combinations activated in the question type classification fully connected network during the identification of the training samples; and the matching degree between the question type identified in the training samples and the labeled type of the training samples meets a set requirement.

[0144] The question type determination module 403 is configured to determine the question type corresponding to the question image according to the target neuron combination and the question type corresponding to the target neuron combination.

[0145] In some embodiments, the question type determination module is specifically configured to:

[0146] If the target neuron combination is not marked with a set mark, determining the question type corresponding to the question image according to the question type corresponding to the target neuron combination; the set mark indicates that the target neuron combination does not meet the set classification requirements;

[0147] If the target neuron combination is marked with a set flag, then:

[0148] The confusion discrimination fully connected network in the multimodal large model is used to continue to identify the question image to obtain a second neuron combination activated in the confusion discrimination fully connected network, the first neuron combination and the second neuron combination are used to generate a third neuron combination, and a target neuron combination matching the third neuron combination is retrieved from the neuron combination library. If a target neuron combination is retrieved, the question type corresponding to the question image is determined based on the question type matched by the retrieved target neuron combination; if not retrieved, another model independent of the multimodal large model is used to identify the question type corresponding to the question image.

[0149] In some embodiments, the device further includes: in response to an event in which any neuron combination in the neuron combination library is marked with a set mark, using the confusion discrimination fully connected network in the multimodal large model to continue to identify the question image corresponding to the neuron combination marked with the set mark, so as to obtain the neuron combination activated by the confusion discrimination fully connected network in the multimodal large model, using the activated neuron combination and the neuron combination marked with the set mark to generate a new neuron combination, and storing the new neuron combination and its corresponding question type in the neuron combination library.

[0150] In some embodiments, the multimodal large model also includes a feature extraction network; the output result of the feature extraction network is input into the question type classification fully connected network; or, when the confusion discrimination fully connected network in the multimodal large model is used to continue to identify the question image, the output result of the feature extraction network is input into the confusion discrimination fully connected network.

[0151] In some embodiments, the apparatus further comprises a step of training the confusion discrimination fully connected network:

[0152] During the training phase of the multimodal large model, different sample features are input into the question type classification fully connected network when the same neuron combination in the question type classification fully connected network is activated;

[0153] The network parameters of the multimodal large model except the confusion discrimination fully connected network are kept unchanged, the different sample features are input into the confusion discrimination fully connected network, and the confusion discrimination fully connected network is used to predict whether the question types of the different sample features are the same to train the confusion discrimination fully connected network.

[0154] In some embodiments, the apparatus further comprises:

[0155] When no target neuron combination matching the first neuron combination is retrieved, using the question type classification fully connected network to output a predicted question type based on the first neuron combination as the question type of the question image;

[0156] The first neuron combination and the predicted question type are recorded in the neuron combination library in correspondence.

[0157] In some embodiments, the apparatus further comprises:

[0158] Recognizing the title text based on the title image;

[0159] The question text and question image are used to perform text search and image search in the existing question library respectively, and when it is determined that the question represented by the question image does not belong to the questions in the existing question library, the input question image is recognized using a multimodal large model.

[0160] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0161] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0162] The embodiment of the present application also provides an electronic device, the structural diagram of the electronic device is as follows Figure 5 As shown, the electronic device 500 includes at least one processor 501, a memory 502 and a bus 503, and the at least one processor 501 is electrically connected to the memory 502; the memory 502 is configured to store at least one computer-executable instruction, and the processor 501 is configured to execute the at least one computer-executable instruction, thereby performing the steps of any question type identification method provided in any embodiment or any optional implementation manner of the present application.

[0163] Furthermore, the processor 501 may be a Field-Programmable Gate Array (FPGA) or other devices with logic processing capabilities, such as a Microcontroller Unit (MCU) or a Central Processing Unit (CPU).

[0164] An embodiment of the present application further provides another readable storage medium storing a computer program, which is used to implement the steps of any question type identification method provided in any embodiment or any optional implementation manner of the present application when executed by a processor.

[0165] The readable storage media provided in the embodiments of the present application include, but are not limited to, any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, the readable storage medium includes any medium that can store or transmit information in a readable form by a device (e.g., a computer).

[0166] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential sequence to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0167] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of specific embodiments of specific inventions. Certain features described in multiple embodiments within this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of the sub-combination.

[0168] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for identifying question types, characterized in that: The method comprises: In the process of recognizing an input question image using the multimodal large model, obtaining a first neuron combination activated in a question type classification fully connected network in the multimodal large model; the first neuron combination includes at least one neuron in the question type classification fully connected network; Retrieving a target neuron combination that matches the first neuron combination from a pre-constructed neuron combination library; the neuron combination library at least includes the corresponding relationship between the question type identified by the training sample used in the training phase of the multimodal large model and the neuron combination activated in the question type classification fully connected network during the identification of the training sample; the matching degree between the question type identified by the training sample and the labeled type of the training sample meets the set requirements; The question type corresponding to the question image is determined according to the target neuron combination and the question type corresponding to the target neuron combination.

2. The method according to claim 1, characterized in that The determining the topic type corresponding to the topic image according to the target neuron combination and the topic type corresponding to the target neuron combination includes: If the target neuron combination is not marked with a set mark, determining the question type corresponding to the question image according to the question type corresponding to the target neuron combination; the set mark indicates that the target neuron combination does not meet the set classification requirements; If the target neuron combination is marked with a set flag, then: The confusion discrimination fully connected network in the multimodal large model is used to continue to identify the question image to obtain a second neuron combination activated in the confusion discrimination fully connected network, the first neuron combination and the second neuron combination are used to generate a third neuron combination, and a target neuron combination matching the third neuron combination is retrieved from the neuron combination library. If a target neuron combination is retrieved, the question type corresponding to the question image is determined based on the question type matched by the retrieved target neuron combination; if not retrieved, another model independent of the multimodal large model is used to identify the question type corresponding to the question image.

3. The method according to claim 2, characterized in that The method further comprises: responding to an event in which any neuron combination in the neuron combination library is marked with a flag; The confusion discrimination fully connected network in the multimodal large model is used to continue to identify the question image corresponding to the neuron combination marked with the set mark, so as to obtain the activated neuron combination of the confusion discrimination fully connected network in the multimodal large model, and a new neuron combination is generated by using the activated neuron combination and the neuron combination marked with the set mark, and the new neuron combination and its corresponding question type are stored in the neuron combination library.

4. The method according to claim 2, characterized in that The multimodal large model also includes a feature extraction network; The output result of the feature extraction network is input into the question type classification fully connected network; Alternatively, when the confusion discrimination fully connected network in the multimodal large model is used to continue to recognize the question image, the output result of the feature extraction network is input into the confusion discrimination fully connected network.

5. The method according to claim 2, characterized in that The method further comprises a step of training the confusion discrimination fully connected network: During the training phase of the multimodal large model, different sample features are input into the question type classification fully connected network when the same neuron combination in the question type classification fully connected network is activated; The network parameters of the multimodal large model except the confusion discrimination fully connected network are kept unchanged, the different sample features are input into the confusion discrimination fully connected network, and the confusion discrimination fully connected network is used to predict whether the question types of the different sample features are the same to train the confusion discrimination fully connected network.

6. The method according to claim 1, characterized in that The method further comprises: When no target neuron combination matching the first neuron combination is retrieved, using the question type classification fully connected network to output a predicted question type based on the first neuron combination as the question type of the question image; The first neuron combination and the predicted question type are recorded in the neuron combination library in correspondence.

7. The method according to claim 1, characterized in that The method further comprises: Recognizing the title text based on the title image; The question text and question image are used to perform text search and image search in the existing question library respectively, and when it is determined that the question represented by the question image does not belong to the questions in the existing question library, the input question image is recognized using a multimodal large model.

8. A device for identifying question types, characterized in that: The device comprises: a neuron combination acquisition module, configured to obtain a first neuron combination activated in a question type classification fully connected network in the multimodal large model during recognition of an input question image using the multimodal large model; the first neuron combination comprising at least one neuron in the question type classification fully connected network; a neuron combination matching module, configured to retrieve a target neuron combination that matches the first neuron combination from a pre-constructed neuron combination library; the neuron combination library at least including the correspondence between the question type identified in the training sample used in the training phase of the multimodal large model and the neuron combinations activated in the question type classification fully connected network during the identification of the training sample; and the degree of matching between the question type identified in the training sample and the labeled type of the training sample meets a set requirement; The question type determination module is used to determine the question type corresponding to the question image according to the target neuron combination and the question type corresponding to the target neuron combination.

9. An electronic device, characterized in that: include: Memory, processor; The memory is used to store computer programs; The processor is configured to call the computer program to implement the method according to any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images