Multi-modal data-driven education large model distribution external detection method and multi-modal data-driven education large model distribution external detection system

Abstract

The invention discloses a multi-modal data-driven education large model distribution external detection method and system, and belongs to the field of multi-modal data processing. According to the method, for a target education scene, a to-be-detected in-distribution category label set and an image data set are obtained, a designed final cue word is input into a large language model to generate an atypical category label set, and an expanded label set is obtained; respectively outputting text features and visual features by a pre-trained text encoder and a pre-trained image encoder, calculating cosine similarity between the visual features and the text features, and constructing a proportional function in a similar normalization form to obtain distribution external scores of the subject images; and finally, decision judgment is carried out, the images outside the distribution are not processed, and a classification data set is formed for the images inside the distribution and is used for executing a classification task in a target education scene. The method can improve the robustness and reliability of the large education model when the large education model is applied to an actual education scene, and has a good application prospect.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal data processing, and in particular to a multimodal data-driven educational large model distribution out-of-detection method and system. Background Art

[0002] With the rapid development of artificial intelligence technology, especially in the field of education, large language models (LLMs) and multimodal large language models (MLLMs) have become important technologies to promote personalized and interactive learning experiences. These models bring rich learning scenarios to education by processing and generating text, sound and visual inputs, such as content innovation, learning support, homework evaluation and classroom feedback. However, while processing and providing rich educational data, these models also face the challenge of out-of-distribution (OOD) data on model performance. Out-of-distribution data refers to input samples that do not meet expectations, such as abnormal homework submitted by students or mistakenly submitted pictures and other atypical behaviors on online education platforms. These samples fall outside the label space of the training dataset, which will lead to unpredictable and often erroneous model behaviors, posing a great threat to the performance of the model. Especially for models used in the field of education, they need to meet the needs of adapting to random and diverse scenarios such as human-computer interaction, teacher-student interaction, and student-student interaction. It is inevitable that unknown educational data (i.e., out-of-distribution data) will appear. Out-of-distribution detection is an important technology for identifying samples outside the distribution of model training data, which is crucial to ensuring the reliability and security of the model. Therefore, out-of-distribution detection techniques for large educational models are crucial to maintaining the quality and trust of educational environments.

[0003] At present, the research on out-of-distribution detection methods mainly focuses on a single modality. For example, AAGOD introduces a data-centric framework to expand the difference between in-distribution and out-of-distribution images, JointEnergy uses energy functions to measure the distribution difference between multi-label images, and SSD proposes a self-supervised out-of-distribution detection framework. These methods can effectively detect out-of-distribution samples based on well-trained in-distribution classifiers. However, they are limited to in-distribution datasets with different label spaces. Some recently proposed methods attempt to design dedicated out-of-distribution detectors for large-scale visual language models (VLMs). For example, ZOC uses a trainable caption generator to create candidate out-of-distribution labels to match out-of-distribution images. The MCM model aims to leverage the power of VLMs to achieve zero-shot out-of-distribution detection by building a text classifier with only in-distribution labels. However, this approach does not work when there are difficult out-of-distribution samples (i.e., OOD samples have a high similarity to in-distribution samples). Therefore, relying solely on in-distribution category labels is not enough to distinguish difficult samples. In addition, although these methods perform well in specific situations, they usually ignore the intrinsic connection between visual images and text labels, which is a significant problem and defect. Summary of the invention

[0004] The purpose of the present invention is to solve the problems existing in the prior art and to provide a multimodal data driven educational large model out-of-distribution detection method and system.

[0005] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a multimodal data-driven out-of-distribution detection method for a large educational model, comprising the following steps:

[0007] S1: obtaining an in-distribution category label set to be detected and an image data set of a target educational subject, processing a common reference prompt word to obtain a final prompt word, inputting the final prompt word into a large language model, generating an atypical category label set by the large language model, and generating an extended label set based on the in-distribution category label set and the atypical category label set; wherein the image data set is composed of question images, and each question image corresponds to an in-distribution category label in the in-distribution category label set;

[0008] S2: Input the category labels in the expanded label set into the pre-trained text encoder to output text features, and input the title images in the image dataset into the pre-trained image encoder to output visual features;

[0009] S3: For a visual feature, calculate the cosine similarity between it and each text feature, use the cosine similarity calculation result as the similarity score between the visual feature and the text feature, use the similarity score between the text feature corresponding to the in-distribution category label set and each visual feature as the first similarity score, use the similarity score between the text feature corresponding to the extended label set and each visual feature as the second similarity score, construct a class-normalized proportional function based on the first similarity score and the second similarity score and use it as the out-of-distribution score of each title image;

[0010] S4: Compare the out-of-distribution score of each question image with the preset distribution domain threshold. If the out-of-distribution score of a question image is greater than the preset distribution domain threshold, the question image is treated as an out-of-distribution image and is not processed. If the out-of-distribution score of a question image is less than or equal to the preset distribution domain threshold, the question image is treated as an in-distribution image. The obtained in-distribution images and their corresponding in-distribution category labels constitute a classification data set, and the education big model is trained on the classification data set. The question images to be classified in the target education subject are input into the trained education big model, and the question type of the question image is output.

[0011] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0012] As a preferred embodiment of the above-mentioned first aspect, in step S1, the generation process of the extended label set is specifically: removing the category labels belonging to the category label set within the distribution from the atypical category label set to obtain a filtered atypical category label set, and merging the filtered atypical category label set with the category label set within the distribution to obtain an extended label set.

[0013] As a preferred embodiment of the first aspect, in step S2, the text encoder adopts a text encoder in a text-image pre-training model based on contrastive learning.

[0014] As a preferred embodiment of the first aspect, in step S2, the image encoder adopts an image encoder in a text-image pre-training model based on contrastive learning.

[0015] As a preferred embodiment of the first aspect, in step S3, the proportional function The specific form is:

[0016]

[0017] In the formula, Represents the number of category labels in the category label set within the distribution; Represents the number of category labels in the filtered atypical category label set; Indicates the number of category labels in the expanded label set; Represents any title image The similarity score between the visual features of and the text features corresponding to the set of category labels within the distribution; Represents any title image The similarity score between the visual features of and the text features corresponding to the expanded label set; The temperature coefficient that controls the smoothness of the proportional function.

[0018] As a preferred embodiment of the first aspect, in step S4, the 95th percentile is selected from all out-of-distribution scores as an initial threshold for reference in setting the distribution domain threshold.

[0019] In a second aspect, the present invention provides a multimodal data driven educational large model out-of-distribution detection system, comprising:

[0020] A data processing module, which obtains an in-distribution category label set to be detected and an image data set of a target educational subject, processes the general reference prompt words to obtain final prompt words, inputs the final prompt words into a large language model, generates an atypical category label set by the large language model, and generates an extended label set based on the in-distribution category label set and the atypical category label set; wherein the image data set is composed of question images, and each question image corresponds to an in-distribution category label in the in-distribution category label set;

[0021] A feature extraction module is used to input the category labels in the extended label set into the pre-trained text encoder to output text features, and input the title images in the image data set into the pre-trained image encoder to output visual features;

[0022] A similarity detection module is used to calculate the cosine similarity between a visual feature and each text feature, use the cosine similarity calculation result as the similarity score between the visual feature and the text feature, use the similarity score between the text feature corresponding to the in-distribution category label set and each visual feature as the first similarity score, use the similarity score between the text feature corresponding to the extended label set and each visual feature as the second similarity score, and construct a proportional function in a class normalized form based on the first similarity score and the second similarity score and use it as the out-of-distribution score of each title image;

[0023] The decision module compares the out-of-distribution score of each question image with the preset distribution domain threshold. If the out-of-distribution score of a question image is greater than the preset distribution domain threshold, the question image is regarded as an out-of-distribution image and is not processed. If the out-of-distribution score of a question image is less than or equal to the preset distribution domain threshold, the question image is regarded as an in-distribution image. The obtained in-distribution images and their corresponding in-distribution category labels constitute a classification data set, and the education big model is trained on the classification data set. The question images to be classified in the target education subject are input into the trained education big model, and the question type of the question image is output.

[0024] In a third aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, can implement the multimodal data-driven educational large model out-of-distribution detection method as described in any one of the schemes of the first aspect above.

[0025] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the multimodal data-driven educational large model out-of-distribution detection method as described in any of the schemes in the first aspect above is implemented.

[0026] In a fifth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0027] The memory is used to store computer programs;

[0028] The processor is used to implement the multimodal data-driven educational large model out-of-distribution detection method as described in any of the schemes of the first aspect above when executing the computer program.

[0029] Compared with the existing out-of-distribution detection methods that rely on labeled data, the present invention has the following beneficial effects:

[0030] The present invention improves the adaptability and robustness of the educational large model to unknown educational data by generating atypical category label sets and multimodal data-driven out-of-distribution detection through zero-sample generation. By utilizing the expert knowledge and reasoning ability of the large language model, potential labels can be generated without additional labeled data, the label space can be expanded, and more accurate out-of-distribution detection can be achieved by combining text features with image features, reducing the dependence on a large amount of labeled data, improving classification accuracy, and making the educational large model more reliable when facing diversified data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the steps of the method of the present invention;

[0032] Figure 2 A schematic diagram of out-of-distribution detection performed by the method of the present invention;

[0033] Figure 3 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0034] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0035] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0036] Multimodal large language models can process multimodal data including text and images, which provides richer contextual information for out-of-distribution detection of educational large models. Inspired by this, the present invention enhances the model's recognition ability for out-of-distribution samples by combining supplementary information from different modalities, thereby improving the robustness and reliability of educational large models when applied to actual educational scenarios.

[0037] like Figure 1 As shown, in a preferred implementation of the present invention, the above multimodal data driven educational large model out-of-distribution detection method includes the following steps S1 to S4. The specific implementation process is described in detail below.

[0038] S1: Obtain an in-distribution category label set to be detected and an image data set of a target educational subject, process the general reference prompt word to obtain a final prompt word, input the final prompt word into a large language model, generate an atypical category label set by the large language model, and generate an extended label set based on the in-distribution category label set and the atypical category label set; wherein the image data set is composed of question images, and each question image corresponds to an in-distribution category label in the in-distribution category label set.

[0039] It should be noted that in step S1, for the two common modal data of text and image in the education model, the present invention first introduces the concept of knowledge-guided prompts. These prompts are intended to use image information to generate out-of-distribution text information to improve the model's recognition ability for unknown inputs. Taking the education scenario as an example, the present invention provides a set of general reference prompt words, which can be fine-tuned or replaced according to specific OOD tasks to adapt to different educational backgrounds and needs.

[0040] The following is an example of a general reference prompt word for a target educational scenario provided by an embodiment of the present invention, wherein the content in {} includes but is not limited to the subject name, homework type or specific teaching topic: "I have compiled a series of pictures of student homework of different {subjects}, covering Unique themes: Please describe the broad categories that these images might fall into, based on their subject matter and content. Categories that are visually similar to these broad categories but do not fit into the existing categories. Please list these for me A new category.” represents the set of labels within the distribution, Represents the total number of categories contained in the label within the distribution.

[0041] The purpose of these prompt words is to inspire the large language model to generate other potential category labels based on the given image information, so that the large language model can better adapt to content that has not been seen before but may appear in the same educational scenario. For example, the prompt words may guide the model to think about different topics in student assignments, and then generate category labels that are visually similar to known categories but belong to different main categories. In this way, the large language model can not only recognize and adapt to known teaching content, but also adjust its generated labels according to example prompt words to guide the detection of out-of-distribution data. In this way, the large language model can not only recognize and adapt to known teaching content, but also adjust the prompts of the large language model according to different educational needs, which helps to improve the large language model's ability to discriminate out-of-distribution data, and can be flexibly adjusted and expanded as needed.

[0042] It should be noted that in the real world, large educational models often face a challenge: they need to identify and process unknown out-of-distribution data that have not been encountered during the training phase. These data may come from different backgrounds, cultures, disciplines, teaching styles or special educational needs, etc. In this case, the model lacks prior knowledge of these unknown categories, and often cannot obtain the exact label information of these out-of-distribution samples, which threatens the robustness and reliability of the model. The present invention uses the expert knowledge and reasoning ability of the large language model to generate a potential atypical category label set, which can help the model better understand and adapt to these unknown and diverse educational data without relying on any actual OOD data, thereby achieving zero-sample training. Specifically, in step S1, the general reference prompt words are fine-tuned or replaced according to the target educational scenario. After obtaining the final prompt words, the final prompt words are output to the large language model, and the large language model generates a series of potential out-of-distribution category labels, and the out-of-distribution category labels constitute an atypical category label set. For example, for the existing data of the online English homework review platform, the label , guiding the large language model to generate a set of atypical category labels for different subjects and question types .in, Both represent out-of-distribution category labels in a set of atypical category labels; Represents the number of out-of-distribution class labels in the set of atypical class labels.

[0043] It should be noted that in step S1, the generation process of the extended label set is specifically as follows: in the atypical category label set, the category labels belonging to the category label set within the distribution are removed to obtain the filtered atypical category label set. , merge the filtered atypical category label set with the in-distribution category label set to obtain the expanded label set .in, Both represent the category labels in the filtered atypical category label set; Represents the number of category labels in the filtered atypical category label set; Both represent category labels in the expanded label set; Represents the number of class labels in the set of class labels within the distribution.

[0044] In this embodiment, when designing the prompt words of the large language model, in order to generate more effective atypical category labels, that is, to make the out-of-distribution category labels as relevant as possible to the in-distribution category labels in the same educational scenario, this type of out-of-distribution data has a higher probability of appearing in the same educational scenario as the in-distribution data, so the potential atypical category labels generated by the large language model have a higher probability of being repeated with the in-distribution category labels. For this reason, this embodiment removes the parts of the atypical category label set that are repeated with the in-distribution category labels by string matching and other methods, ensuring that the remaining atypical category labels have no direct relationship with the in-distribution category labels, that is, It can be seen that the filtering step is crucial to maintain the classification accuracy of the model and avoid confusion. Finally, the filtered atypical category label set is merged with the in-distribution category label set to obtain an extended label set. This extended label set contains more diverse category labels and can more comprehensively cover various themes and topics that may appear in specific educational scenarios. In this way, the model will have more choices and richer information when performing OOD detection, thereby improving its ability to discriminate unknown samples.

[0045] S2: Input the category labels in the expanded label set into the pre-trained text encoder and output text features , input the title image in the image dataset into the pre-trained image encoder and output the visual features .in, Respectively represent Visual features of class labels; Respectively represent Visual features of each title image.

[0046] It should be noted that the current out-of-distribution detection methods often ignore the connection between the title image and the text label, and do not make full use of the information of each modality. The present invention enhances the model's ability to represent samples by combining features from different modalities, thereby improving the model's ability to understand and classify samples. Specifically, in step S2, the present invention uses a text encoder and an image encoder to extract features from the text and the title image, respectively, in order to perform more accurate OOD detection. The text encoder is responsible for processing the text labels of educational content, while the image encoder processes education-related title images, such as charts, experimental photos, student photos, etc., and then extracts key visual elements in the title image, such as graphics, symbols, and objects, to obtain visual features. In this embodiment, the text encoder and the image encoder respectively use the text encoder and image encoder in the text-image pre-training (CLIP) model based on contrastive learning.

[0047] S3: For a visual feature, calculate the cosine similarity between it and each text feature, and use the cosine similarity calculation result as the similarity score between the visual feature and the text feature, use the similarity score between the text feature corresponding to the in-distribution category label set and each visual feature as the first similarity score, use the similarity score between the text feature corresponding to the extended label set and each visual feature as the second similarity score, and construct a class-normalized proportional function based on the first similarity score and the second similarity score and use it as the out-of-distribution score of each title image.

[0048] It should be noted that in step S3, the calculation result of the cosine similarity between the visual features and each text feature can be used to represent the similarity between a title image in the image data set and each category label in the extended label set. Cosine similarity is a value between -1 and 1, which is used to measure the angle between two vectors, thereby reflecting their similarity. It is calculated as follows:

[0049]

[0050]

[0051]

[0052] in, Indicates Visual features of the title image With Text features of category labels The cosine similarity between .

[0053] In this embodiment, if Figure 2 As shown in the figure, the above cosine similarity calculation result is used as the similarity score between the visual feature and the text feature, and the similarity matrix is ​​constructed from the similarity score. Each element in the similarity matrix is ​​a similarity score, the row index corresponds to the visual feature of each title image, and the column index corresponds to each text feature. On this basis, by normalizing the above similarity matrix, a proportional function similar to the normalized form is designed. , take any title image The similarity with the in-distribution category label and the similarity with the atypical category label are combined to generate a score that can reflect whether the sample is an out-of-distribution image, namely the out-of-distribution score, which comprehensively considers the information between different modalities and different distributions.

[0054] It should be noted that in step S3, the above-mentioned proportional function The specific form is:

[0055]

[0056] In the formula, Represents any title image The similarity score between the visual features of and the text features corresponding to the category label set in the distribution, that is, the first similarity score; Represents any title image The similarity score between the visual features of and the text features corresponding to the expanded tag set, i.e., the second similarity score; The temperature coefficient that controls the smoothness of the proportional function.

[0057] It can be seen that the out-of-distribution score provided by the present invention has the following characteristics: if the similarity between the question image and the in-distribution category label is high, the out-of-distribution score will also be high, indicating that the question image is more likely to be an in-distribution image; if the similarity with the atypical category label is high, the out-of-distribution score will also be low, indicating that the question image is more likely to be an out-of-distribution image. This calculation method ensures the accuracy of the model's evaluation of the difference in data distribution domains in different educational scenarios.

[0058] S4: Compare the out-of-distribution score of each question image with the preset distribution domain threshold. If the out-of-distribution score of a question image is greater than the preset distribution domain threshold, the question image is treated as an out-of-distribution image and is not processed. If the out-of-distribution score of a question image is less than or equal to the preset distribution domain threshold, the question image is treated as an in-distribution image. The obtained in-distribution images and their corresponding in-distribution category labels constitute a classification data set, and the education big model is trained on the classification data set. The question images to be classified in the target education subject are input into the trained education big model, and the question type of the question image is output.

[0059] It should be noted that in step S4, the 95th percentile is selected as the initial threshold among all out-of-distribution scores for reference in setting the distribution domain threshold.

[0060] In the present invention, the purpose of out-of-distribution detection is to ensure that the model has the ability to discriminate new samples outside its training data. The model calculates the distance between the input question image (such as student homework, test answers or interactive data) and each category label in the extended label set to obtain an out-of-distribution score. The out-of-distribution score of the question image is compared with the set distribution domain threshold to decide whether the input question image is an in-distribution image or an out-of-distribution image. Therefore, it is necessary to dynamically adjust the distribution domain threshold of the out-of-distribution detection according to the changes in the educational scene, which is an important guarantee for maintaining the sensitivity and accuracy of the large education model. The obtained in-distribution images constitute a classification data set, and the classification task in the target education scene is performed to ensure the robustness and reliability of the large education model when applied to actual education scenes.

[0061] In this embodiment, a statistical method is provided to analyze the distribution of out-of-distribution scores of in-distribution images (normal educational data), and the 95th percentile is selected as the initial threshold. :

[0062]

[0063] in, is the set of out-of-distribution scores in ascending order; By changing the quantile, based on the change in the area score under the receiver operating characteristic curve, it is decided whether to optimize the initial threshold to determine the distribution domain threshold of normal education data in this education scenario.

[0064] It should be noted that, through the method of the present invention, the educational model can more comprehensively evaluate the relationship between the input sample and the known category label, thereby improving the ability to distinguish unknown or unusual samples. In step S4, the out-of-distribution score of each question image is compared with the preset distribution domain threshold. The decision method provided in step S4 of the present invention can be expressed as a decision function :

[0065]

[0066] In this embodiment, taking English as the target education subject as an example, the image data set used in step S1 is obtained by the online English homework review platform. However, this image data set is not entirely composed of the question images of the English subject, and may be mixed with question images of other subjects. The above process of the present invention is used to perform out-of-distribution detection on each question image in the image data set, and the question images of the English subject are regarded as in-distribution images, while the question images of the mathematics subject are regarded as out-of-distribution images. If it is an out-of-distribution image, further prediction (such as homework statistics, test scores or emotional state prediction) is rejected. However, for some images identified as out-of-distribution images, the education big model can provide customized educational intervention suggestions, which may include recommending additional learning materials, personalized exercises or further discussions with teachers. If it is judged to be an in-distribution image, the original downstream task is continued. For example, the obtained in-distribution images constitute a classification data set, which is composed only of the question images of the English subject and the in-distribution category labels. The education big model can be trained to complete the classification task in the target education scenario.

[0067] It should be noted that, in the present invention, the educational big model refers to a big model used in the field of education, and the technicians in this field can select the corresponding model structure according to actual needs to perform the corresponding downstream tasks. This embodiment specifically adopts the ViT-B / 16 model as the educational big model, which is used to implement the classification task of the question image and output the question type contained in the question image. Among them, the implementation method of the ViT-B / 16 model belongs to the prior art and will not be repeated.

[0068] In order to better demonstrate the specific implementation and technical effects of the present invention, the multimodal data-driven educational large model out-of-distribution detection method shown in steps S1 to S4 in the above preferred implementation method is applied to a specific example.

[0069] Example

[0070] The specific implementation process of the multimodal data-driven educational large model out-of-distribution detection method used in this embodiment is as described above and will not be repeated here.

[0071] In order to demonstrate the technical effect of this embodiment, the present invention is verified under the setting of ImageNet-1K as the in-distribution dataset and iNaturalist, Places, and SUN as the out-of-distribution datasets. Among them, the iNaturalist dataset contains images of plants and animals, specifically 675,170 training images and verification images from 5,089 natural fine-grained categories, and the categories of these images belong to 13 major categories, including plants, insects, birds, mammals, etc. The SUN dataset and the Places dataset both contain natural landscape images. The SUN dataset is a dataset for scene understanding, which covers 130,519 images in 899 categories; the Places dataset has 10 million scene classification images, labeled with scene semantic categories, including a large number of types of environmental types in the world. Therefore, compared with other OOD datasets, the iNaturalist dataset has the greatest difference in category terms with the in-distribution category dataset. The SUN dataset and the Places dataset usually contain multiple elements from the natural world, beyond their annotation basis facts, which makes them more easily confused with the in-distribution category data. The embodiments of the present invention use these three data sets as out-of-distribution data sets, and then the performance verification of the out-of-distribution detection method is relatively reliable. The experimental results are shown in Table 1. In Table 1, AUROC is the area under the receiver operating characteristic curve, and the larger the value, the better the performance; FPR95 is the false positive rate of OOD data when the true positive rate of in-distribution data is 95%, and the smaller the value, the better the performance. Energy, CLIPN and MCM are classic models for OOD detection. The implementation methods of these models belong to the prior art and will not be repeated here. The results in Table 1 show that the method of the present invention has advantages over these classic models in both AUROC and FPR95 indicators.

[0072] Table 1. Experimental comparison results of this embodiment and three basic models

[0073] In summary, the out-of-distribution detection method based on a large model proposed in the present invention helps the educational large model to identify input samples that do not meet expectations, thereby improving the robustness and reliability of the educational large model when applied to actual educational scenarios. For example, in an online education platform, this method can be used to automatically evaluate and analyze homework submitted by students, so as to timely discover and quickly provide targeted feedback. In a smart education system, the method of the present invention can also be applied to learning path recommendations, personalized learning, and teaching content optimization, so as to provide students and teachers with better educational experience and support.

[0074] It should also be noted that the multimodal data-driven educational model out-of-distribution detection method in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a multimodal data-driven educational model out-of-distribution detection system corresponding to the multimodal data-driven educational model out-of-distribution detection method provided in the above embodiment, such as Figure 3 As shown, it includes:

[0075] A data processing module, which obtains an in-distribution category label set to be detected and an image data set of a target educational subject, processes the general reference prompt words to obtain final prompt words, inputs the final prompt words into a large language model, generates an atypical category label set by the large language model, and generates an extended label set based on the in-distribution category label set and the atypical category label set; wherein the image data set is composed of question images, and each question image corresponds to an in-distribution category label in the in-distribution category label set;

[0076] A feature extraction module is used to input the category labels in the extended label set into the pre-trained text encoder to output text features, and input the title images in the image data set into the pre-trained image encoder to output visual features;

[0077] A similarity detection module is used to calculate the cosine similarity between a visual feature and each text feature, use the cosine similarity calculation result as the similarity score between the visual feature and the text feature, use the similarity score between the text feature corresponding to the in-distribution category label set and each visual feature as the first similarity score, use the similarity score between the text feature corresponding to the extended label set and each visual feature as the second similarity score, and construct a proportional function in a class normalized form based on the first similarity score and the second similarity score and use it as the out-of-distribution score of each title image;

[0078] The decision module compares the out-of-distribution score of each question image with the preset distribution domain threshold. If the out-of-distribution score of a question image is greater than the preset distribution domain threshold, the question image is regarded as an out-of-distribution image and is not processed. If the out-of-distribution score of a question image is less than or equal to the preset distribution domain threshold, the question image is regarded as an in-distribution image. The obtained in-distribution images and their corresponding in-distribution category labels constitute a classification data set, and the education big model is trained on the classification data set. The question images to be classified in the target education subject are input into the trained education big model, and the question type of the question image is output.

[0079] It is understandable that the multimodal data-driven educational model out-of-distribution detection method described in S1 to S4 above can be substantially implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the multimodal data-driven educational model out-of-distribution detection method provided in the above embodiment, which includes a computer program / instruction. When the computer program / instruction is executed by a processor, the multimodal data-driven educational model out-of-distribution detection method described in the above embodiment can be implemented.

[0080] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the multimodal data driven educational large model distribution out-of-distribution detection method provided in the above embodiment, which includes a memory and a processor;

[0081] The memory is used to store computer programs;

[0082] The processor is used to implement the multimodal data-driven educational large model out-of-distribution detection method in the above-mentioned embodiment when executing the computer program.

[0083] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0084] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the multimodal data-driven educational large model distribution out-of-detection method provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by the processor, it can implement the multimodal data-driven educational large model distribution out-of-detection method in the above embodiment.

[0085] It is understandable that the above storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. The storage medium may also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc., which can store program codes.

[0086] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0087] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0088] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A multimodal data driven out-of-distribution detection method for educational large models, characterized in that: The following steps are involved: S1: obtaining an in-distribution category label set to be detected and an image data set of a target educational subject, processing a common reference prompt word to obtain a final prompt word, inputting the final prompt word into a large language model, generating an atypical category label set by the large language model, and generating an extended label set based on the in-distribution category label set and the atypical category label set; wherein the image data set is composed of question images, and each question image corresponds to an in-distribution category label in the in-distribution category label set; S2: Input the category labels in the expanded label set into the pre-trained text encoder to output text features, and input the title images in the image dataset into the pre-trained image encoder to output visual features; S3: For a visual feature, calculate the cosine similarity between it and each text feature, use the cosine similarity calculation result as the similarity score between the visual feature and the text feature, use the similarity score between the text feature corresponding to the in-distribution category label set and each visual feature as the first similarity score, use the similarity score between the text feature corresponding to the extended label set and each visual feature as the second similarity score, construct a class-normalized proportional function based on the first similarity score and the second similarity score and use it as the out-of-distribution score of each title image; S4: Compare the out-of-distribution score of each question image with the preset distribution domain threshold. If the out-of-distribution score of a question image is greater than the preset distribution domain threshold, the question image is treated as an out-of-distribution image and is not processed. If the out-of-distribution score of a question image is less than or equal to the preset distribution domain threshold, the question image is treated as an in-distribution image. The obtained in-distribution images and their corresponding in-distribution category labels constitute a classification data set, and the education big model is trained on the classification data set. The question images to be classified in the target education subject are input into the trained education big model, and the question type of the question image is output.

2. A multimodal data driven educational large model out-of-distribution detection method as claimed in claim 1, characterized in that: In step S1, the generation process of the extended label set is specifically as follows: in the atypical category label set, the category labels belonging to the in-distribution category label set are removed to obtain a filtered atypical category label set, and the filtered atypical category label set is merged with the in-distribution category label set to obtain an extended label set.

3. A multimodal data driven educational large model out-of-distribution detection method as claimed in claim 1, characterized in that: In step S2, the text encoder uses a text encoder in a text-image pre-training model based on contrastive learning.

4. A multimodal data driven educational large model out-of-distribution detection method as claimed in claim 1, characterized in that: In step S2, the image encoder uses an image encoder in a text-image pre-training model based on contrastive learning.

5. A multimodal data driven educational large model out-of-distribution detection method as claimed in claim 1, characterized in that: In step S3, the proportional function The specific form is: ； In the formula, Represents the number of category labels in the category label set within the distribution; Represents the number of category labels in the filtered atypical category label set; Indicates the number of category labels in the expanded label set; Represents any title image The similarity score between the visual features of and the text features corresponding to the set of category labels within the distribution; Represents any title image The similarity score between the visual features of and the text features corresponding to the expanded label set; The temperature coefficient that controls the smoothness of the proportional function.

6. A multimodal data driven educational large model out-of-distribution detection method as claimed in claim 1, characterized in that: In step S4, the 95th percentile is selected from all out-of-distribution scores as the initial threshold value, which is used as a reference when setting the distribution domain threshold value.

7. A multimodal data driven educational large model out-of-distribution detection system, characterized in that: include: A data processing module, which obtains an in-distribution category label set to be detected and an image data set of a target educational subject, processes the general reference prompt words to obtain final prompt words, inputs the final prompt words into a large language model, generates an atypical category label set by the large language model, and generates an extended label set based on the in-distribution category label set and the atypical category label set; wherein the image data set is composed of question images, and each question image corresponds to an in-distribution category label in the in-distribution category label set; A feature extraction module is used to input the category labels in the extended label set into the pre-trained text encoder to output text features, and input the title images in the image data set into the pre-trained image encoder to output visual features; A similarity detection module is used to calculate the cosine similarity between a visual feature and each text feature, use the cosine similarity calculation result as the similarity score between the visual feature and the text feature, use the similarity score between the text feature corresponding to the in-distribution category label set and each visual feature as the first similarity score, use the similarity score between the text feature corresponding to the extended label set and each visual feature as the second similarity score, and construct a proportional function in a class normalized form based on the first similarity score and the second similarity score and use it as the out-of-distribution score of each title image; The decision module compares the out-of-distribution score of each question image with the preset distribution domain threshold. If the out-of-distribution score of a question image is greater than the preset distribution domain threshold, the question image is regarded as an out-of-distribution image and is not processed. If the out-of-distribution score of a question image is less than or equal to the preset distribution domain threshold, the question image is regarded as an in-distribution image. The obtained in-distribution images and their corresponding in-distribution category labels constitute a classification data set, and the education big model is trained on the classification data set. The question images to be classified in the target education subject are input into the trained education big model, and the question type of the question image is output.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, it can implement the multimodal data-driven educational large model distribution detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the multimodal data-driven educational large model out-of-distribution detection method as described in any one of claims 1 to 6 is implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the multimodal data-driven educational large model out-of-distribution detection method as described in any one of claims 1 to 6 when executing the computer program.

Images