Image-text matching method, electronic device, and storage medium

Abstract

The application provides a kind of text matching method, electronic equipment and storage medium, wherein the method comprises: obtaining test question text;Knowledge point entity recognition is carried out on test question text, and the knowledge point entity in test question text and the entity relationship of knowledge point entity are obtained;Test question type and test question difficulty of test question text are obtained by carrying out test question type classification and difficulty classification on test question text;Based on the knowledge point entity in test question text and the entity relationship of knowledge point entity, test question type and test question difficulty of test question text, determine the illustration requirement information of test question text;Based on the illustration requirement information, and the illustration description information of each candidate illustration, determine the target illustration matched with test question text from each candidate illustration.The method, electronic equipment and storage medium provided by the application can improve the efficiency of text matching while ensuring the reliability and accuracy of text matching.

Description

Technical Field

[0001] This invention relates to the field of educational information technology, and in particular to a text-image matching method, electronic device, and storage medium. Background Technology

[0002] With the digitalization and intelligentization of educational resources, the combination of text and images has become the mainstream form of educational content production.

[0003] Currently, matching suitable illustrations to test questions usually requires professional artists to design the illustrations manually, which is time-consuming and costly. Although there are solutions for automated illustration matching, most of them are based on keywords in the test questions. Due to a lack of deep semantic understanding of the test questions, the matched illustrations often fail to match the actual content of the test questions or even contain conceptual errors.

[0004] Therefore, how to optimize the accuracy and reliability of image-text matching while ensuring its efficiency remains a pressing issue in the field of educational informatization. Summary of the Invention

[0005] This invention provides a text-image matching method, electronic device, and storage medium to address the shortcomings of related technologies in achieving both text-image matching efficiency and accuracy.

[0006] This invention provides an image-text matching method, comprising:

[0007] Obtain the test question text;

[0008] The test question text is subjected to knowledge point entity recognition to obtain the knowledge point entities in the test question text and the entity relationships of the knowledge point entities. The test question text is classified by question type and difficulty to obtain the test question type and test question difficulty of the test question text.

[0009] Based on the knowledge point entities and entity relationships in the test question text, the test question type and difficulty of the test question text, the illustration requirement information of the test question text is determined;

[0010] Based on the illustration requirement information and the illustration description information of each candidate illustration, a target illustration that matches the test question text is determined from the candidate illustrations.

[0011] According to a text-image matching method provided by the present invention, the step of performing knowledge point entity recognition on the test question text to obtain the knowledge point entities in the test question text and the entity relationships of the knowledge point entities includes:

[0012] Entity recognition is performed on the test question text to obtain the entities in the test question text;

[0013] The entity is mapped to a knowledge graph to obtain the knowledge point entity and the graph node of the knowledge point entity in the knowledge graph;

[0014] Relationship path reasoning is performed on the graph nodes of the knowledge point entity in the knowledge graph to obtain the entity relationship of the knowledge point entity.

[0015] According to a text-image matching method provided by the present invention, the entity recognition of the test question text includes:

[0016] Encode the word features of each word in the test text, wherein the word features include the word vector, position vector and segmentation vector of the word;

[0017] Based on the dependencies between the lexical features of each word in the test text, the contextual features of each word in the test text are determined.

[0018] Entity recognition is performed based on the contextual features of each word in the test text.

[0019] According to a text-image matching method provided by the present invention, the step of classifying the test question text by question type includes:

[0020] The test texts are classified into macro-level question types, subject-specific categories, and assessment objectives.

[0021] According to a text-image matching method provided by the present invention, the illustration description information includes at least one of the subject knowledge points, visual features, and teaching attributes of the candidate illustration, and the illustration description information is generated based on a large language model.

[0022] According to a text-image matching method provided by the present invention, the step of determining a target illustration that matches the test question text from the candidate illustrations based on the illustration requirement information and the illustration description information of each candidate illustration includes:

[0023] Based on the similarity between the semantic requirements of the illustration requirement information and the semantic requirements of the illustration description information of each candidate illustration, a target illustration that matches the test question text is determined from the candidate illustrations.

[0024] According to a text-image matching method provided by the present invention, the step of determining the target illustration that matches the test question text from the candidate illustrations based on the similarity between the demand semantics of the illustration demand information and the illustration semantics of the illustration description information of each candidate illustration includes:

[0025] Based on the similarity between the requirement semantics of the illustration requirement information and the illustration semantics of the illustration description information of each candidate illustration, and the similarity between the text semantics of the test question text and the image features of each candidate illustration, a target illustration matching the test question text is determined from the candidate illustrations.

[0026] The image-text matching method provided by the present invention further includes:

[0027] If no target illustration matching the question text is found among the candidate illustrations, a visual prompt is determined based on the illustration requirement information, and a target illustration matching the question text is generated based on the visual prompt.

[0028] The image-text matching method provided by the present invention further includes:

[0029] Based on the question type of the test question text and the device type of the display device, a layout strategy is determined;

[0030] Based on the layout strategy, the test question text and the target illustration are displayed.

[0031] The present invention also provides an image-text matching device, comprising the following modules:

[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image-text matching method as described above.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image-text matching method as described above.

[0034] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image-text matching method as described above.

[0035] The image-text matching method, electronic device, and storage medium provided by this invention perform knowledge point entity recognition, reminder classification, and difficulty classification on the test question text, realizing deep semantic analysis of the test question text, thereby determining the illustration requirement information of the test question text. Based on the illustration requirement information and the illustration description information of each candidate illustration, the target illustration that matches the test question text is determined from each candidate illustration, thereby improving the efficiency of image-text matching while ensuring the reliability and accuracy of image-text matching. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is one of the flowcharts illustrating the image-text matching method provided by the present invention.

[0038] Figure 2 This is a flowchart illustrating the process of obtaining test question text provided by the present invention.

[0039] Figure 3 This is a schematic diagram illustrating the process of determining the required information for the illustrations provided by the present invention.

[0040] Figure 4 This is a schematic diagram illustrating the matching process between test question text and candidate illustrations provided by the present invention.

[0041] Figure 5 This is a schematic diagram of the process for generating the target illustration provided by the present invention.

[0042] Figure 6 This is a flowchart illustrating the graphic and text layout process provided by the present invention.

[0043] Figure 7 This is the second flowchart of the image-text matching method provided by the present invention.

[0044] Figure 8 This is a schematic diagram of the image-text matching system provided by the present invention.

[0045] Figure 9 This is a schematic diagram of the image-text matching device provided by the present invention.

[0046] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0048] Currently, matching suitable illustrations to test questions typically requires professional artists to design them manually. This process is not only time-consuming and labor-intensive, with the production cycle for a single illustration potentially taking several hours, but also sees labor costs increase exponentially as the question bank expands. More seriously, manual illustration is insufficient to meet the real-time illustration needs of online education platforms with their massive question volumes, becoming a bottleneck restricting the large-scale production of educational content.

[0049] To improve the efficiency of image-text matching, the following three matching schemes have been proposed:

[0050] One approach is keyword-based matching. This method uses TF-IDF (term frequency–inverse document frequency) or Word2Vec to extract keywords from the questions, matches these keywords with preset tags in an illustration library, and then uses a rule engine to combine the text and images. This approach has three main drawbacks: Firstly, at the semantic understanding level, it lacks a deep understanding of the questions' meanings, making it unable to distinguish the subject-specific connotations of words. The matched illustrations may be disconnected from the actual content of the questions, or even contain conceptual errors. For example, keyword matching might ignore the difference between "force" in physics and its everyday usage; matching the keyword "triangle" might fail to differentiate the application scenarios of different types of triangles, resulting in inappropriate visual examples. Secondly, at the contextual understanding level, it's difficult to distinguish the specific meanings of the same keyword in different questions. For example, in a math question, "circle" might represent a geometric figure or a function graph. Thirdly, at the adaptability level, keywords cannot reflect the learning stage of the questions, and therefore cannot adjust the complexity of illustrations to suit different learning stages.

[0051] This mechanical matching fails to grasp the underlying knowledge system and teaching intent of the questions, resulting in a disconnect between the illustrations and the content. For example, in math word problems, "speed" might refer to average speed or instantaneous speed, but the system cannot distinguish between them; in history problems, "revolution" might correspond to different historical events, but the accompanying illustrations are often identical. This superficial matching severely impacts teaching effectiveness and the learning experience.

[0052] Another approach is based on template rules: This involves establishing a subject-specific template library (e.g., mathematics, physics, chemistry), pre-setting fixed illustration templates for each subject's questions, and using regular expressions to extract question parameters for filling in the templates. This approach has significant drawbacks: First, maintaining the template library is costly, requiring manual creation of corresponding templates for each new question type; second, flexibility is severely limited, as it cannot automatically adjust illustrations based on question parameters; and finally, it lacks innovation and struggles to meet the illustration needs of interdisciplinary comprehensive questions.

[0053] Another approach is based on machine learning: it uses traditional deep learning models such as LSTM (Long Short-Term Memory) or CNN (Convolutional Neural Network) to train on labeled question-image pairs and borrows from collaborative filtering recommendation methods to achieve image-text matching. This approach also has significant drawbacks: firstly, it heavily relies on labeled data, requiring a large number of manually labeled training samples; secondly, it faces the cold start problem, performing poorly when encountering new knowledge points in the questions; thirdly, the model's decision-making process lacks interpretability, making it difficult to guarantee the accuracy of the educational content; and most importantly, the alignment between the text and image feature spaces is not precise enough.

[0054] In summary, current image-text matching solutions mostly employ rule engines or simple machine learning models, which have limitations:

[0055] First, relying on manually defined, rigid rules makes it difficult to cover complex and ever-changing educational scenarios. In educational settings, questions of different subjects and difficulty levels require illustrations of varying complexity and style. Current matching schemes lack flexible adaptation mechanisms and fail to consider cognitive patterns and subject characteristics, making it impossible to adjust the complexity of illustrations or differentiate between the visualization needs of arts and sciences based on learning stages.

[0056] For example, science problems require precise formulas and diagrams, while humanities problems require contextualized illustrations; elementary school students need simple diagrams, while high school students need more professional analytical diagrams. However, current image-text matching schemes either use uniform templates or require manually preset rules, and cannot automatically adjust according to the characteristics of the subject and cognitive patterns, resulting in illustrations that are either too simple or too complex, thus losing their pedagogical value.

[0057] Secondly, the lack of understanding of the deeper semantics and teaching intentions of the test questions results in limited educational value of the generated text-image combinations. Furthermore, due to the inherent modal differences between text-based test questions and visual illustrations, traditional keyword- or template-based methods struggle to establish accurate cross-modal associations, leading to a semantic gap in the matching of test questions and illustrations. This insufficient multimodal alignment accuracy directly impacts the effectiveness of text-image matching.

[0058] Third, the system lacks scalability, making it difficult to adapt to emerging educational formats such as AR (Augmented Reality) / VR (Virtual Reality) teaching and adaptive learning. Each new question type requires a corresponding template, or the model faces a cold start problem when encountering new knowledge point questions. These factors severely restrict the expanded application of the image-text matching solution, leading to difficulties in large-scale adaptation and slow iteration, thus affecting the quality and intelligent development of educational products.

[0059] To address the above problems, embodiments of the present invention provide a text-image matching method. Figure 1 This is one of the flowcharts illustrating the image-text matching method provided by the present invention, such as... Figure 1 As shown, the method includes:

[0060] Step 110: Obtain the test question text.

[0061] Here, the test question text refers to the text form of the test question that needs to be matched with the image. The test question text can be the text of the questions in the test question.

[0062] The acquisition of test question text can be achieved by directly extracting test question text from a test question bank, or by directly receiving test question text input by the user. Alternatively, it can be achieved by extracting raw test question data from a test question bank, or by receiving raw test question data input by the user, and then standardizing, cleaning, and structuring the raw test question data to transform it into structured test question text. This embodiment of the invention does not specifically limit the specific methods used. It should be noted that this embodiment of the invention supports input of raw test question data via text, voice, handwriting, images, and other input channels.

[0063] The standardization and cleaning of the original test data includes removing extra spaces, garbled characters, irrelevant symbols, correcting spelling errors, and standardizing terminology. Symbol filtering can be implemented using regular expressions, standardized terminology replacement can be achieved using an educational terminology database, and correcting spelling errors can be done using context-aware methods. For example, newline characters "\n" can be removed, and unit symbols can be standardized (e.g., converting "Ω" to "ohm"). Furthermore, special content such as mathematical formulas and chemical equations can be separately marked and converted to Markdown format.

[0064] Standardized cleaning can accommodate the irregularities in the original test data input by users, while avoiding information loss or misconversion, thus ensuring the robustness of image-text matching.

[0065] Furthermore, for cases where the original test data is multimodal data such as speech, handwriting, and images, standardization and cleaning of the original test data can also include converting multimodal test data to text modality using modality conversion tools. For example, a domain-optimized ASR (Automatic Speech Recognition) system based on the Conformer model can be used to convert speech modality data; a handwriting recognition system integrating the StrokeNet neural network can be used to convert handwriting modality data; and the MinerU open-source tool can be used to convert image modality data.

[0066] Subsequently, the standardized and cleaned test data is structured, which can form structured text. This structured text can include the original text, cleaned text, and modality types, and can be regarded as the test text for subsequent applications.

[0067] In some embodiments, Figure 2 This is a flowchart illustrating the process of obtaining test question text provided by the present invention, such as... Figure 2 As shown, the process of obtaining test question text can be as follows: obtain the original test question data, determine the modality of the original test question data as a supportable data modality, perform modality transformation on the original test question data, perform standardization and cleaning on the test question data transformed into text modality, and finally generate the structured output of the test question data, that is, obtain the structured test question text.

[0068] For example, the process of converting raw test data into structured test text can be formalized as follows:

[0069]

[0070] in, This is the original test data. For structured test question text, This involves a multi-level standardized cleaning and structuring process from raw test data to structured test text. This process yields high-quality structured output, ensuring the image-text matching method is compatible with different input methods and remains robust, providing a reliable foundation for subsequent semantic analysis of the test text.

[0071] Among them, the original test data ,in For the first The dataset contains test questions, where N is the number of original test questions. Through multi-level processing, the test questions can be standardized and structured. , For the first A structured test question text.

[0072] For example, the original test data could be:

[0073] A circuit has a resistor of 10Ω. When a voltage of 20V is applied across its terminals...

[0074] (1) Calculate the current in the circuit according to Ohm's law;

[0075] (2) If this circuit is to generate a 4A current, what voltage needs to be adjusted? (An equation needs to be written and solved).

[0076] The test text obtained after standardized cleaning and structuring can be:

[0077] {“Original Text”:“A circuit has a resistance of 10Ω. When a voltage of 20V is applied across its terminals, (1) calculate the current in the circuit according to Ohm's Law; (2) to generate a current of 4A through the circuit, how many volts should the voltage be adjusted? (Equations need to be set up and solved)”,“Clean Text”:“A circuit has a resistance of 10 ohms. When a voltage of 20V is applied across its terminals, (1) calculate the current in the circuit according to Ohm's Law; (2) to generate a current of 4A through the circuit, how many volts should the voltage be adjusted? (Equations need to be set up and solved)”,

[0078] "Modal type": "text"}.

[0079] Step 120: Perform knowledge point entity recognition on the test question text to obtain the knowledge point entities in the test question text and the entity relationships of the knowledge point entities. Perform question type classification and difficulty classification on the test question text to obtain the question type and question difficulty of the test question text.

[0080] Specifically, after obtaining the test question text, semantic analysis can be performed on it. Here, semantic analysis of the test question text can be divided into three categories: knowledge point entity recognition, question type classification, and difficulty classification.

[0081] The knowledge point entity recognition function identifies knowledge point entities within the test question text and extracts their relationships. This recognition can be implemented using a pre-trained NLP (Natural Language Processing) model. Here, knowledge point entities refer to the knowledge points reflected in the test question text, such as mathematical formulas, physical laws, or historical events. These entities can be associated with definitions in a knowledge graph. Furthermore, for interdisciplinary questions, multiple domain knowledge points can be labeled simultaneously to ensure the accuracy of subsequent illustrations. The resulting knowledge point entities and their relationships can be stored in a structured format, including, for example, the knowledge point name, its subject, and its contextual relationship within the test question text.

[0082] Question type recognition is used to classify question types in test texts at a fine-grained level. This classification process can combine rule engines and machine learning to distinguish the question types and assessment objectives within the test texts. By obtaining the question types from the test texts, thematic relevance in subsequent text-image matching can be ensured; for example, matching diagrams for geometry questions and flowcharts for experimental question matching operations.

[0083] Difficulty classification is used to assess the difficulty of test texts. The assessment of test difficulty can be multi-dimensional, such as based on textual features, logical complexity, and reference to educational standards. For example, test texts classified as basic questions can be labeled L1 (memory / comprehension), while those classified as comprehensive questions can be labeled L3 (application / analysis). The level of difficulty directly affects the complexity of illustrations.

[0084] Step 130: Based on the knowledge point entities in the test question text and the entity relationships of the knowledge point entities, the test question type and the test question difficulty of the test question text, determine the illustration requirement information of the test question text.

[0085] Specifically, after obtaining the results of knowledge point entity recognition, question type classification, and difficulty classification for the test question text, the results of the above knowledge point entity recognition, question type classification, and difficulty classification can be combined to generate the illustration requirement information for the test question text.

[0086] Here, the illustration requirements information for the test question text can include parameters for the illustrations that need to be matched with the test question text, such as the content type, complexity, and interactivity of the required illustrations. This illustration requirements information provides clear constraints for subsequent text-image matching, thereby ensuring the pedagogical suitability of the text and images.

[0087] Furthermore, in some embodiments, the illustration requirement information may include not only parameters specifying the illustrations to be matched with the test question text, but also the results of the aforementioned knowledge point entity recognition, question type classification, and difficulty classification. In other words, the illustration requirement information reflects not only the test question text's need for matched illustrations, but also the semantics of the test question text itself, facilitating more accurate text-image matching. For example, the illustration requirement information may be structured JSON data including four-dimensional tags: knowledge point, question type, question difficulty, and illustration requirement.

[0088] For example, after obtaining the results of knowledge point entity recognition, question type classification, and difficulty classification of the test question text, the illustration complexity parameter can be determined based on the test question difficulty in the above results, combined with Bloom's cognitive level and grade level requirements. The resulting illustration requirement information is stored as a set of metadata in the form of structured JSON data. The illustration requirement information may include:

[0089] Question ID: Used to uniquely identify the question. It can adopt the coding rule of "subject-serial number" to enable question tracking and internal indexing.

[0090] Macro-level question types: These generally include multiple choice, fill-in-the-blank, true / false, experimental, or problem-solving questions.

[0091] Parameters: Records entity information extracted from the test question text, along with corresponding descriptions.

[0092] Core Knowledge Points: This section contains object information for multiple knowledge point entities. Each entity's object information may include fields such as concept, subject, semantic role, and related concepts. Specifically, the concept identifies the name of the knowledge point entity involved in the test text, ensuring that subsequent illustrations accurately match the teaching content; the subject precisely locates the subject and branch to which the knowledge point entity belongs, thus influencing the professionalism and presentation style of the illustrations; the semantic role defines the function of the knowledge point entity in the test text, thereby determining the priority and presentation method of the illustrations, with common values ​​including key points of examination, calculation tools, and background knowledge; and related concepts list the relevant concepts upon which the problem-solving depends, ensuring the completeness of the illustrations and the coherence of the teaching.

[0093] Interdisciplinary connections: The main subject areas involved in this question are used to trigger interdisciplinary visualization strategies.

[0094] Difficulty Levels: Four levels of difficulty can be used, specifically divided into L1-L4. The difficulty level determines the complexity and detail of the illustrations.

[0095] Cognitive levels: Based on Bloom's taxonomy of educational objectives, common values ​​include memorization, comprehension, application, analysis, evaluation, and creation.

[0096] For example, the structured output of the illustration requirement information for a test question text can be represented in the following form:

[0097] {

[0098] "Problem ID": "PHYS-002",

[0099] "Macroeconomic Question Types": "Short Answer Questions"

[0100] "Parameter":[{"Entity":"Resistance","Value":"10", "Unit": "Ohms","Role":"Known Conditions"},{"Entity":"Voltage","Value":"20","Unit": "Volts", "Role":"Known Conditions"},

[0101] {"Entity":"Current","Value":"To be determined","Unit": "Ampere", "Role":"Solution objective (1)"},{"Entity":"Current","Value":"4","Unit": "Ampere","Role":"Given objective (2)"},

[0102] {"Entity":"Voltage","Value":"To be determined","Unit": "Volts", "Role":"Solution objective (2)"},],

[0103] "Core Knowledge Points": [

[0104] {

[0105] Concept: Ohm's Law

[0106] Subject: Physics - Electricity

[0107] "Semantic Role": "Key Exam Focus",

[0108] Related concepts: ["Current", "Voltage", "Resistance"]

[0109] },

[0110] {

[0111] Concept: Algebraic equations

[0112] Subject: Mathematics - Algebra

[0113] Semantic Role: "Computational Tool"

[0114] }

[0115] ],

[0116] "Interdisciplinary connections": ["Physics", "Mathematics"],

[0117] Difficulty Level: L2

[0118] "Cognitive Level": "Application"

[0119] }

[0120] In this embodiment of the invention, step 120 uses NLP technology to deeply analyze the semantic content of the test question text, and step 130 determines the illustration requirements of the test question text based on its semantic content. The processes of steps 120 and 130 can be formalized as follows:

[0121]

[0122] in, For structured test question text, For structured illustration requirements information, In order to execute the process, The execution process accurately extracts knowledge point entities and associates them with the knowledge graph, identifies question types and difficulties, and dynamically generates parameterized illustration requirement information. The application of steps 120 and 130 breaks through the limitations of traditional keyword matching. The illustration requirement information obtained from this process starts from the deep semantics of the test text, improves the relevance of text and images, and supports continuous learning and updating, providing an intelligent analysis foundation for educational text and image matching.

[0123] Figure 3 This is a flowchart illustrating the process of determining the illustration requirements information provided by the present invention, such as... Figure 3 As shown, semantic analysis can be performed in parallel on the test text. Specifically, entity recognition, knowledge graph query, and relation extraction can be performed on the test text to determine the knowledge point entities and their relationships. In addition, question type and difficulty classification can be performed on the test text to determine the question type and difficulty. Combining the above methods, the required information for illustrations can be determined.

[0124] Step 140: Based on the illustration requirement information and the illustration description information of each candidate illustration, determine the target illustration that matches the test question text from the candidate illustrations.

[0125] Specifically, candidate illustrations are pre-collected educational illustrations, which can be obtained by acquiring resources such as educational test questions and other educational illustrations. Acquisition of candidate illustrations supports methods such as scanning and digitizing textbooks and capturing open-source resources. Furthermore, after acquiring candidate illustrations, they can be standardized.

[0126] Furthermore, illustration description information can be pre-labeled for each candidate illustration. This illustration description information can be understood as a semantic tag for the candidate illustration, describing multi-dimensional information such as subject knowledge points, visual features, and teaching attributes, in order to facilitate intelligent classification and accurate retrieval of candidate illustrations.

[0127] After obtaining the illustration requirements for the test question text, the illustration requirements information can be matched with the illustration description information of each candidate illustration. This allows for the selection of the candidate illustration that best matches the test question text; this is denoted as the target illustration. Furthermore, the selection of the target illustration can be achieved by calculating the similarity between the illustration requirements information and the illustration description information of each candidate illustration. That is, the candidate illustration with the highest similarity can be selected as the target illustration. This achieves accurate text-image matching in educational scenarios, balancing retrieval efficiency and matching accuracy.

[0128] The process of selecting target illustrations for test texts based on illustration requirement information can be formalized as follows:

[0129]

[0130] in, Information required for illustrations in the test question text. The target illustration for the test question text. To streamline the process of matching test question text with candidate illustrations, selecting target illustrations based on illustration requirement information can improve the efficiency and accuracy of text-image matching while reducing material search time.

[0131] Furthermore, the quality of the illustration library containing the candidate illustrations can be continuously optimized through a dynamic update mechanism, thereby ensuring that the matched target illustrations not only meet the requirements of teaching science, but also adapt to the educational needs of different grade levels and regions, providing strong visual support for personalized teaching.

[0132] In the method provided in this embodiment of the invention, knowledge point entity recognition, reminder classification, and difficulty classification are performed on the test question text to achieve deep semantic analysis of the test question text, thereby determining the illustration requirement information of the test question text. Based on the illustration requirement information and the illustration description information of each candidate illustration, the target illustration that matches the test question text is determined from each candidate illustration, thereby improving the efficiency of image-text matching while ensuring the reliability and accuracy of image-text matching.

[0133] Based on the above embodiments, in step 120, the step of performing knowledge point entity recognition on the test question text to obtain the knowledge point entities in the test question text and the entity relationships of the knowledge point entities includes:

[0134] Entity recognition is performed on the test question text to obtain the entities in the test question text;

[0135] The entity is mapped to a knowledge graph to obtain the knowledge point entity and the graph node of the knowledge point entity in the knowledge graph;

[0136] Relationship path reasoning is performed on the graph nodes of the knowledge point entity in the knowledge graph to obtain the entity relationship of the knowledge point entity.

[0137] Specifically, for the test question text, the BERT (Bidirectional Encoder Representations from Transformers) model or other models that can be used for command entity recognition can be used to identify entities in the test question text and label information such as the type of the entity. The information labeled as the type of entity here can include the type of entity in the test question, such as a parameter, law, or method, and can also include the value, unit, and role of the entity in the test question, such as the role being known condition or solution objective.

[0138] For example, given a test question text, the following entities and their information can be extracted:

[0139] {

[0140] "Number of parameters": [

[0141] {"Entity": "Resistance", "Value": "10", "Unit": "Ohms", "Role": "Known Conditions"},

[0142] {"Entity": "Voltage", "Value": "20", "Unit": "Volts", "Role": "Known Conditions"},

[0143] {"Entity": "Current", "Value": "To be determined", "Unit": "Ampere", "Role": "Solution objective (1)"},

[0144] {"Entity": "Current", "Value": "4", "Unit": "Ampere", "Role": "Given Target (2)"},

[0145] {"Entity": "Voltage", "Value": "To be determined", "Unit": "Volt", "Role": "Solution objective (2)"},

[0146] ],

[0147] "Laws / Methods": ["Ohm's Law", "Algebraic Equations"]

[0148] }

[0149] After extracting entities from the test question text, these entities can be matched against a knowledge graph. Here, the knowledge graph used for matching can be a knowledge graph from various disciplines. By matching entities with the knowledge graph, entities can be mapped into the knowledge graph; that is, the corresponding graph node for each entity can be found within the knowledge graph. It can be understood that for an entity with a matching graph node in the knowledge graph, that is, based on its existence and semantic correctness as a knowledge point, it can be considered a knowledge point entity. The knowledge point entities identified in this way can eliminate potential ambiguities in cross-disciplinary terminology.

[0150] For example, the text representation of the entity can first be converted into a 768-dimensional vector. Through trainable matrices Reduce the dimensionality of this vector to the knowledge graph embedding space:

[0151]

[0152] in, It is a text representation of entities reduced to the knowledge graph embedding space, with an output dimension of 256, consistent with the embedding dimension of graph nodes in the knowledge graph; ReLU is an activation function used to enhance sparsity and filter noise. It is a trainable matrix.

[0153] Based on this, we can calculate Cosine similarity between the embedding representations of the nodes in the knowledge graph and the embedded representations of the nodes:

[0154]

[0155] In the formula, for Embedded representation of the i-th graph node in the knowledge graph The cosine similarity between them. N is the total number of graph nodes in the knowledge graph.

[0156] The similarity between each entity and a node in the knowledge graph can be obtained, and the three nodes with the highest similarity are selected as knowledge point nodes. For example, in the physics-electricity knowledge graph, the entity Ohm's law has a similarity of sim=0.95; in the physics-fundamentals knowledge graph, the entity current has a similarity of sim=0.82; and in the mathematics-algebra knowledge graph, the entity linear equation has a similarity of sim=0.76.

[0157] After mapping entities to a knowledge graph, relational path reasoning can be performed on the graph nodes based on the knowledge point entities' positions within the knowledge graph and the connections between these nodes. This quantifies the logical relationships between knowledge point entities, allowing the extraction of entity relationships from the knowledge graph, thus obtaining the entity relationships of the knowledge point entities. Specifically, the entity relationships of knowledge point entities can be represented as entities in the test text that are associated with the knowledge point entities.

[0158] Specifically, based on a pre-defined relationship matrix, relational path reasoning can be performed on graph nodes to obtain the path probabilities between entities. , can be represented as:

[0159]

[0160] in, For the sigmoid function, This is a matrix set of predefined educational-specific relationship types. For example, subject-specific relationships may include: belonging to (physics-electricity), interdisciplinary connections, etc.; teaching logic relationships may include: prerequisite knowledge (current → resistance), deepening concepts, etc.; cognitive relationships may include: L1 → L2 difficulty progression, memorization → application level improvement, etc. Each type of relationship... Corresponding to a trainable relation matrix .

[0161] Multi-hop path search can be used to reason about the relationship paths between entity mappings to nodes in the knowledge graph. During this process, educational constraints can be set to filter out invalid paths; for example, "historical event → physics formula" can be filtered out as an invalid path. For each path that meets the requirements, the optimal path can be determined by calculating a confidence score.

[0162]

[0163] in, For educational weight.

[0164] By integrating educational theory into a relational path search system, the teaching logic can be made computable. It can automatically identify entity association paths that conform to the laws of cognitive development. For example, in physics problems, it can accurately connect the teaching chain of "Ohm's Law → Series Circuit → Resistance Calculation" to avoid reasoning results that are beyond the scope of the curriculum or have logical breaks.

[0165] Once the relationship path is obtained, the relationships between entities can be extracted from it. Specifically, relationship extraction can be performed based on the starting and ending entities in the relationship path.

[0166] First, by modeling the joint probability of entity-relationship relationships, the logical connections between knowledge points can be quantified, ensuring that the relationships extracted from entities conform to the subject's logic. Specifically, this can be achieved by using the vectors of the starting entities extracted by BERT. and endpoint entity vector In combination with the context Through the weight matrix Calculate the relationship conditional probability distribution :

[0167]

[0168] Then, by leveraging the global structural constraints of the knowledge graph, ambiguities arising from local text extraction can be avoided, thus enabling the establishment of entity relationships. Specifically, this can be achieved by utilizing predefined entity embeddings within the knowledge graph. And relation matrix Calculate triples The reasonableness score is calculated, with lower scores indicating a higher likelihood of a valid relationship. This leads to the relationship between the starting and ending entities in the relationship path.

[0169] In addition, classification weight matrix can also be used. This allows for hierarchical classification and semantic role analysis of entities, thereby clarifying the functional positioning of knowledge point entities within the test text and guiding subsequent visualization strategies. Specifically, entities can be... Embedded vector Mapped to semantic role category c:

[0170]

[0171] For example, the semantic role of "Ohm's Law" is classified as "a key point of examination", while the semantic role of "algebraic equations" is classified as "a computational tool".

[0172] Therefore, after completing the knowledge point entity recognition for the test text, the following structured information can be obtained to reflect the knowledge point entities, entity relationships, and semantic roles of knowledge point recognition in the test text. Entity relationships can be represented as "associated concepts," reflecting the logical dependencies between entities in the test text:

[0173] {

[0174] "Core Knowledge Points": [

[0175] {

[0176] Concept: Ohm's Law

[0177] Subject: Physics

[0178] "Semantic Role": "Key Exam Focus",

[0179] Related concepts: ["Current", "Voltage", "Resistance"]

[0180] },

[0181] {

[0182] Concept: Algebraic equations

[0183] Subject: Mathematics

[0184] Semantic Role: "Computational Tool"

[0185] } ]

[0187] }

[0188] Based on any of the above embodiments, in step 120, the entity recognition of the test question text includes:

[0189] Encode the word features of each word in the test text, wherein the word features include the word vector, position vector and segmentation vector of the word;

[0190] Based on the dependencies between the lexical features of each word in the test text, the contextual features of each word in the test text are determined.

[0191] Entity recognition is performed based on the contextual features of each word in the test text.

[0192] Specifically, when extracting entities from the test text, a segmented coding and hierarchical representation fusion strategy based on the test structure was adopted, aiming to more accurately capture the semantic information of knowledge points contained in each component of the test text.

[0193] First, when representing the test text using vectors, each token in the test text can be encoded separately, thereby obtaining the lexical features of each token. Here, the lexical features of each token in the test text can be represented in the following form:

[0194]

[0195] Among them, the word sequence of the test text It consists of n lexical units, where the lexical feature of the j-th lexical unit is... It is word vector Position vector and piecewise vectors The summation of . Where This indicates that keywords in the test question text are encoded, such as "function" or "equation"; This reflects the sequence number of the j-th word element in the sequence, thus preserving the word order information, such as "calculate the derivative" which depends on the context. This reflects the paragraph number of the j-th word in the test text. By superimposing the above three types of vectors, the test text is transformed into a processable numerical vector. Among them, It can be obtained through BERT encoding.

[0196] It is understandable that lexical features combining word vectors, position vectors, and segmentation vectors can reflect not only the meaning of the word itself and its position in the sequence, but also the paragraph level of the word within the test text. Therefore, in entity recognition based on lexical features, it is possible to better capture the semantic information of the knowledge points contained in each component of the test from the perspective of paragraph level. Since word vectors are obtained through BERT encoding, this hierarchical lexical feature combining segmentation vectors can be denoted as Hie-BERT.

[0197] After obtaining the lexical features of each word, semantic encoding can be performed on the serialized test text, that is, extracting the contextual features of each word in the test text. Here, the extraction of contextual features can be achieved through a multi-layer Transformer encoder, and this process can be represented as:

[0198]

[0199] In the formula, It is the first The contextual features of each word element. In the process of extracting the contextual features of each word element in the test text, a multi-head attention mechanism can be used to calculate the dependencies between word elements:

[0200]

[0201] Among them, Q, V, and K are respectively composed of word sequences from the test text. Obtained by linear transformation, This is the scaling factor. For example, in the math exam question "Find the function..." In the term "derivative," the word "derivative" requires attention to the preceding "function" and "derivative." The attention mechanism can automatically learn this long-distance dependency, ensuring that the "derivative" is correctly classified as a mathematical term.

[0202] After extracting the contextual features of each word, entities in the test text can be decoded based on these features, thus enabling the structured extraction of knowledge points. This is based on the contextual features of each word. The probability distribution for entity recognition can be expressed as:

[0203]

[0204] in, and These are classification layer parameters. This indicates the predicted entity recognition result.

[0205] In this embodiment of the invention, the contextual features of each word can not only achieve entity recognition, but also be used for entity aggregation and subject adaptation, thereby transforming educational test questions into computable knowledge point data.

[0206] Based on any of the above embodiments, in step 120, classifying the test question text by question type includes:

[0207] The test texts are classified into macro-level question types, subject-specific categories, and assessment objectives.

[0208] Specifically, the question types can be categorized based on the test text, which can be achieved from three dimensions: macro-level question type categorization, subject-specific categorization, and assessment target categorization.

[0209] Macro-level question classification refers to determining the macro-level question type category to which the test text belongs, such as multiple choice, fill-in-the-blank, true / false, experimental, or problem-solving questions. For example, the BERT model can be combined with a rule engine to achieve accurate macro-level question type classification for test texts through semantic understanding and structured features.

[0210] For example, the BERT model can be used to classify test question texts. Encoding is performed to obtain the context vector of the test question text. Then calculate the probability distribution of macro-level questions. :

[0211]

[0212] in, and The parameters of the BERT model for question type classification. express This is a macro-level question. The probability of.

[0213] Subject-specific classification refers to determining the subject to which a test question belongs and the type of subject within that subject. For example, the subject-specific classification result for "finding the area of ​​a circle" in the test question is: Mathematics - Geometry; the subject-specific classification result for "calculating electric field strength" is: Physics - Electromagnetism; and the subject-specific classification result for "balancing chemical equations" is: Chemistry - Chemical Reactions.

[0214] For example, the BERT model for subject-specific classification can be used to classify test texts by subject and output the probability distribution of subject-specific classification. :

[0215]

[0216] in, and The parameters of the BERT model for subject-specific classification; Represents the regular feature vector; express Subject-specific specialization The probability of.

[0217] The classification of assessment objectives refers to determining the assessment objectives of the test questions, such as memorization, calculation, and comprehensive application.

[0218] For example, a multilayer perceptron can be used to classify the target and output the probability distribution of the target. :

[0219]

[0220] in, This is a multilayer perceptron prediction model that integrates the semantic vectors of the questions extracted by the BERT model. Mathematical expression complexity Question text length Take this information as input, and you can get the output. ,express Subject to investigation The probability of.

[0221] Based on any of the above embodiments, in step 120, the test question text is classified by difficulty, including:

[0222] Based on the statistical and semantic features of the test text, the test text is classified into difficulty levels.

[0223] The statistical features of the test text are obtained by TF-IDF extraction, and can be specifically represented as follows: Furthermore, the semantic features of the test text can be extracted based on the BERT model, and these semantic features can be represented as follows: .

[0224] The statistical and semantic features of the test question text can be integrated to form the feature vector of the test question text. For example, it can be represented as:

[0225]

[0226] The feature vectors of the test text obtained by this fusion are enhanced by applying TF-IDF to increase the weights on complex terms, and the activation values ​​output by BERT are applied to make the distribution of the feature vectors closer to higher-order cognitive dimensions.

[0227] Then, the feature vectors of the test question text can be used to classify the difficulty of the test questions:

[0228]

[0229] in, and The parameters for the difficulty classification model. This indicates the difficulty level of the test questions. Here, the difficulty level reflects the logical complexity of the questions.

[0230] The difficulty levels categorized here correspond to Bloom's hierarchy of cognitive levels. The difficulty levels can be divided into L1 to L4, and the corresponding Bloom cognitive levels can be obtained from the table below.

[0231]

[0232] Based on any of the above embodiments, the method further includes:

[0233] Collect multi-source data to construct an illustration library, which is used to store candidate illustrations.

[0234] Specifically, the illustration library can be a multimodal educational illustration library, including various forms such as static images and dynamic charts. The data sources for the illustration library can be multiple, including open-source educational resources, digitized illustrations from professional textbooks, and content uploaded by teacher users. All illustrations undergo standardization to ensure uniformity in resolution, color mode, and file format, and a hierarchical storage index is established. Simultaneously, copyright compliance reviews are implemented, and commercially restricted resources are replaced or licensed to form a safe and usable educational illustration library.

[0235] Furthermore, the process of collecting data to build an illustration library can be represented as:

[0236]

[0237] in, This represents a combination of test questions and images. This indicates the data source space, meaning that candidate illustrations can be collected from scanned copies of textbooks, open-source resources, and user-uploaded content. This is the original data space.

[0238] In educational settings, scanned copies of textbooks Using scan resolution constraints Ensure the accuracy of illustration conversion; user-uploaded content Ensuring image usability through quality filters; open-source resources. A distributed crawler based on the Scrapy framework was built to implement a targeted crawling strategy for educational websites.

[0239] Furthermore, data standardization can be performed on the acquired images: First, define the image resampling function. , to image Converting to a standardized resolution output that meets the needs of educational scenarios can be mathematically expressed as:

[0240]

[0241] in, The target resolution is dynamically calculated, combining an absolute threshold of 300 dpi with relative size constraints, to ensure the clarity requirements in teaching scenarios. , This represents the height and width of the image.

[0242] Then, the image is transformed through a non-linear mapping. Color data converted to standard sRGB space To ensure the accurate reproduction of key visual elements in teaching scenarios, its mathematical expression can be defined as:

[0243]

[0244] in, For images The result of color conversion.

[0245] Finally, define the format conversion function. Achieve unified conversion of illustration formats, among which Education-specific formatting strategy:

[0246]

[0247] Therefore, after acquiring images from multiple data sources, the acquired images can be standardized, and the standardized images can be stored as candidate illustrations in the illustration library.

[0248] Based on any of the above embodiments, the illustration description information includes at least one of the subject knowledge points, visual features, and teaching attributes of the candidate illustration, and the illustration description information is generated based on a large language model.

[0249] Specifically, for the collected candidate illustrations, prompt words can be constructed to guide a large language model to generate illustration description information for the candidate illustrations. Furthermore, prompt words can be constructed to guide the large language model to generate illustration description information for the candidate illustrations from at least one aspect of the candidate illustrations' subject knowledge points, visual features, and pedagogical attributes.

[0250] Among them, subject knowledge points refer to the subject to which the candidate illustration is adapted, as well as the knowledge points under that subject. Visual features can describe the core objects in the candidate illustration, the layout of the candidate illustration, color and other information. Teaching attributes can include information such as the difficulty level of the test questions and the grade level to which the candidate illustration is adapted.

[0251] For example, you can set the following prompts to retrieve illustration description information for candidate illustrations:

[0252] You are an intelligent annotation system for educational resources. Please perform multimodal analysis on the given illustrations and complete the following tasks:

[0253] [User Issue]:

[0254] 'Educational Illustrations'

[0255] [Your task]:

[0256] 1. Visual feature extraction: Describe the core objects, layout, colors, etc. in the image.

[0257] 2. Text information recognition: Extract text, formulas, or annotations from the image.

[0258] 3. Teaching tag generation:

[0259] Subject categories: Mathematics / Physics / Chemistry / Biology / History, etc.

[0260] Key concepts: Specific concepts (such as 'Pythagorean theorem' and 'redox reaction')

[0261] Difficulty Levels: L1 (Basic Memory) - L3 (Advanced Application)

[0262] Educational stages (primary / middle / high school / university)

[0263] 4. Semantic retrieval optimization: Generate 3-5 natural language query examples (e.g., 'Experiment diagram of Newton's first law in high school physics').

[0264] 5. Confidence Assessment and Recommendations: Fields with low confidence levels (<80%) should be manually reviewed.

[0265] [Output format example]:

[0266] {

[0267] "visual_analysis": ["...", "..."], / / Visual feature description

[0268] "text_analysis": ["...", "..."], / / Text / formula in the image

[0269] "educational_tags": {

[0270] "subject": "physics", / / subject

[0271] "topic": "Newton's Second Law", / / Key Points

[0272] "difficulty": "L2", / / Difficulty (L1-L3)

[0273] "grade_level": "high school", / / grade level

[0274] "interactivity": "interactive" / / Interactivity

[0275] },

[0276] "search_queries": [ / / Search query examples]

[0277] "Dynamic Demonstration of F=ma Experiment in High School Physics"

[0278] Interactive simulation diagram of Newton's Second Law

[0279] ],

[0280] "confidence_scores": { / / Label confidence score (0-1)

[0281] "subject": 0.95,

[0282] "topic": 0.88,

[0283] "difficulty": 0.75

[0284] },

[0285] "expert_review_suggestions": [ / / Fields requiring expert review]

[0286] "difficulty" ]

[0288] }

[0289] Based on similar cue words, large language models can infer from the input candidate illustrations and cue words, and then output illustration description information for the candidate illustrations. This process can be formalized as follows:

[0290]

[0291] in, Construct a constructor for the prompt words. For candidate illustrations, This refers to the illustration description information output by a large-scale language model, which can include subject knowledge points, visual features, teaching attributes, etc. Furthermore, the large-scale language model can also output confidence assessments and suggestions for the above illustration description information, to help determine whether to subsequently submit the illustration description information to experts for verification, thereby ensuring the accuracy of the illustration description information.

[0292] Alternatively, the process of annotating candidate illustrations with descriptive information can be semi-automated. For example, the associated descriptive text of the candidate illustrations can be obtained first, and then parsed using an NLP model to obtain basic tags for the candidate illustrations, such as subject and knowledge point. Furthermore, visual tags for the candidate illustrations can be extracted using a CV (Computer Vision) model, such as color composition and structural features. Subsequently, educational experts can manually annotate teaching attribute tags, such as applicable grade level and interaction requirements. The illustration descriptive information of the candidate illustrations, as a type of tag, adopts a hierarchical ontology design, supporting flexible retrieval from coarse-grained to fine-grained.

[0293] Based on any of the above embodiments, in step 140, determining the target illustration that matches the test question text from the candidate illustrations based on the illustration requirement information and the illustration description information of each candidate illustration includes:

[0294] Based on the similarity between the semantic requirements of the illustration requirement information and the semantic requirements of the illustration description information of each candidate illustration, a target illustration that matches the test question text is determined from the candidate illustrations.

[0295] Specifically, to achieve matching between the test question text and candidate illustrations, semantic features can be extracted from the illustration requirement information and the illustration description information respectively. In this embodiment of the invention, the semantic features of the illustration requirement information are denoted as requirement semantics, and the semantic features of the illustration description information are denoted as illustration semantics.

[0296] The acquisition of demand semantics and illustration semantics can be achieved through a pre-trained semantic extraction model, such as the BGE-small model, which can be used to extract illustration description information for each candidate illustration in the illustration library. Encode it and map it to A vector space of dimension 1. The vectorization process of extracting the semantics of illustrations based on illustration description information can be formalized as follows:

[0297]

[0298] in, Illustration description information for candidate illustrations The dense semantic vector, which is the illustration semantics of the candidate illustration.

[0299] Embedding models like BGE-small are trained on massive corpora. These models effectively capture the deep semantic features of text, making semantically similar illustration descriptions closer to each other in the generated illustration semantic vector space. For example, the illustration descriptions of two candidate illustrations... and The semantic similarity score between them can usually be obtained through their illustrated semantics. and cosine similarity To measure:

[0300]

[0301] in, The L2 norm (Euclidean length) of a vector. The value of is between -1 and 1, and for non-negative vectors it is usually between 0 and 1. The closer the value is to 1, the more semantically similar it is. Due to the superiority of the BGE-small model, even and The length differences between them are significant, and this similarity score can reliably reflect the true semantic closeness.

[0302] Furthermore, demand semantics can be obtained based on an NLP encoder or a model of this type.

[0303] Based on this, by quantitatively analyzing the similarity between the semantic requirements of illustrations in the test question text and the semantics of illustration descriptions in each candidate illustration, high-precision automatic matching of test question text and illustrations can be achieved. For example, the candidate illustration with the highest similarity can be used as the target illustration to match the test question text.

[0304] Furthermore, in the process of retrieving target illustrations that match the test question text from the illustration library based on similarity, a hybrid retrieval strategy can be introduced, combining sparse retrieval for rapid initial screening and dense retrieval for fine ranking, thus balancing efficiency and accuracy.

[0305] Based on any of the above embodiments, step 140, which involves determining the target illustration matching the test question text from the candidate illustrations based on the similarity between the requirement semantics of the illustration requirement information and the illustration semantics of the illustration description information of each candidate illustration, includes:

[0306] Based on the similarity between the requirement semantics of the illustration requirement information and the illustration semantics of the illustration description information of each candidate illustration, and the similarity between the text semantics of the test question text and the image features of each candidate illustration, a target illustration matching the test question text is determined from the candidate illustrations.

[0307] Specifically, when matching test text with candidate illustrations, not only can semantic matching be performed using illustration requirement information from the test text and illustration description information from the candidate illustrations, but cross-modal matching can also be performed directly using the test text and candidate illustrations.

[0308] Here, cross-modal matching between the test question text and candidate illustrations can be achieved based on a cross-modal attention mechanism. First, semantic features can be extracted separately for the test question text and candidate illustrations. Here, the semantic features of the test question text are denoted as text semantics, and the semantic features of the candidate illustrations are denoted as image features. For example, the BERT model can be used to encode the test question text to obtain text semantics. And using CNN models to analyze image features of candidate illustrations Extraction can be represented by the following formula:

[0309]

[0310] in, ( ) is a function used to extract semantics from text, CNN ( ) is a function used to extract image features. Additionally, the BERT-encoded test text can be processed using the CLIP (Contrastive Language-Image Pre-training) model. and CNN encoding of candidate illustrations Mapped to a shared semantic space.

[0311] Based on this, similarity calculations can be performed on the text semantics and image features in the shared semantic space.

[0312] Alternatively, before performing similarity calculations, a Knowledge Graph Revised Attention Rectification Module (KARM) can be introduced. KARM identifies and reinforces cross-modal interactions related to core educational concepts by querying the educational knowledge graph. Specifically, this can be achieved through text semantics. The system identifies key educational entities / concepts and uses them to query the educational knowledge graph to obtain subgraphs containing related concepts, attributes, and relationships. The information of subgraph G is encoded into a knowledge context vector. This knowledge context vector serves as a guiding signal and can participate in calculating cross-modal attention between text semantics and image features. This process can be briefly formulated as follows:

[0313]

[0314] in, and It is textual semantics and image features that have been modified by a knowledge-guided attention mechanism, making educational semantics more prominent.

[0315] Subsequently, suitable candidate illustrations can be matched to the test text by calculating the similarity between the textual semantics of the test text and the image features of the candidate illustrations. The similarity calculation here can be expressed by the formula:

[0316]

[0317] in, The weighting of terms for education. These are preset parameters.

[0318] Based on this, the target illustration can be determined from the candidate illustrations by combining the similarity between the semantics of the demand and the illustration description information of each candidate illustration, as well as the similarity between the semantics of the text and the image features of each candidate illustration. For example, the two types of similarity can be weighted and summed, and the target illustration can be determined from the candidate illustrations based on the weighted summed similarity.

[0319] In this embodiment of the invention, matching based on an attention mechanism ensures a high degree of consistency between the subject matter and details of the target illustration and the test question text. Specifically, a cross-modal attention mechanism based on the Transformer architecture can be used to achieve fine-grained alignment between the test question and the illustration. On the text side, higher attention weights can be assigned to key terms, while on the visual side, the semantic core area of ​​the illustration is focused. For complex questions, multi-head attention is used to match different modules such as the experimental setup diagram, data recording table, and result curve graph, and the final weighted summation of the matching score is then performed.

[0320] Figure 4 This is a schematic diagram illustrating the matching process between test question text and candidate illustrations provided by the present invention, such as... Figure 4 As shown, multimodal data can be pre-collected and standardized to facilitate the collection of candidate illustrations and the construction of an illustration library. Furthermore, illustration description information is annotated for the candidate illustrations. After obtaining the test question text, the similarity between the illustration requirement information of the test question text and the illustration description information of the candidate illustrations can be calculated. Then, the test question text and candidate illustrations are matched based on a cross-module attention mechanism. Combining these two methods, candidate illustrations that match the test question text are selected from the illustration library as target illustrations.

[0321] Based on any of the above embodiments, after step 140, the method further includes:

[0322] If no target illustration matching the question text is found among the candidate illustrations, a visual prompt is determined based on the illustration requirement information, and a target illustration matching the question text is generated based on the visual prompt.

[0323] Specifically, for cases where the target illustration cannot be matched from the candidate illustrations, a generative artificial intelligence model can be used to generate a target illustration that matches the test question text. This process can be formalized as follows:

[0324]

[0325] in, Information required for illustrations in the test question text. Target illustrations that match the test question text. This process enables generative AI-generated educational illustrations. Based on a generative artificial intelligence model, it addresses the core pain points of incomplete illustration resource coverage and outdated updates in traditional educational content production through intelligent generation and dynamic optimization mechanisms.

[0326] Figure 5 This is a schematic diagram of the generation process of the target illustration provided by the present invention, such as... Figure 5 As shown, the process can be divided into stages such as demand analysis and triggering, semantic-visual parameterization conversion, generation model adaptation and optimization, and dynamic parameter adjustment.

[0327] The demand analysis and triggering mechanism aims to initiate the proactive generation process of the target illustration. Determining that no target illustration among the candidate illustrations matches the question text can serve as a condition for triggering target illustration generation. This determination can be made by ensuring that the similarity between the question text and each candidate illustration is below a preset threshold.

[0328] For example, for a test question text to be matched, the test question text can be compared with each candidate illustration in the illustration library. Similarity score If the similarity score Below the preset threshold If no matching candidate illustration is found in the illustration library, the automatic generation process is triggered. Similarity threshold triggering conditions. This can be expressed as a formula:

[0329]

[0330] Semantic-visual parametric transformation aims to convert the illustration requirements of structured test texts into visual instructions executable by artificial intelligence models, thereby achieving a lossless transformation from abstract knowledge points to precise teaching illustrations. Specifically, it can convert the illustration requirements of JSON-formatted test texts into parametric prompts for generative artificial intelligence models, i.e., obtain visual prompts.

[0331] In practice, the illustration requirements of the test text in JSON format can be parsed to extract key teaching elements such as physics formulas, mathematical parameters, and interactive components. Then, multimodal alignment technology can be used to map these key teaching elements into control parameters for visual generation and combine them into visual prompts.

[0332] Generative model adaptation and optimization aims to adapt and optimize generative artificial intelligence models so that they can output illustrations that fit the test text based on the input visual prompts.

[0333] For example, a generative AI model could be a Stable Diffusion (SD) model, which can generate precise and standardized illustrations based on input visual cues. Specifically, a pre-trained SD model can be fine-tuned using LoRA based on educational data (test text-illustration pairs) to adapt it to academic illustration styles while retaining its original generative capabilities. Furthermore, the output style can be controlled by adding special trigger words. The LoRA fine-tuning parameters are as follows:

[0334] train_config:

[0335] pretrained_model: "stabilityai / stable-diffusion-2-1-base"

[0336] dataset: "edu_dataset_v1"

[0337] lora_rank: 64

[0338] batch_size: 8

[0339] learning_rate: 1e-4

[0340] text_encoder_lr: 5e-5

[0341] Steps: 5000

[0342] trigger_word: "edu_diagram

[0343] By fine-tuning the SD model, it can dynamically associate numerical variables in the question text with image elements in the illustrations. For the SD model, ControlNet constraints can be incorporated to generate structures, such as edge detection to ensure correct graphic proportions. Furthermore, variable values ​​can be extracted using the text parsing module and embedded into visual prompts. Variable-visual binding can be described as:

[0344]

[0345] in, To generate an image; Generate functions for the SD model; Visual prompts; Canny edge map extracted for ControlNet; It is random noise; τ is the edge detection operator; M is the expected edge template; τ is the tolerance threshold.

[0346] Building upon this, the CLIP model can be used to evaluate the semantic consistency between the generated target illustrations and the test text. Based on the semantic consistency evaluation results, the templates for visual cues and the fine-tuning strategy of the SD model can be iteratively optimized. This ensures that the generated target illustrations both conform to educational norms and can flexibly respond to changes in variables. The iterative optimization index is semantic alignment loss. This can be expressed as:

[0347]

[0348] in, For CLIP image encoder; A text encoder for CLIP; Let be the cosine similarity.

[0349] The end-to-end inference process based on a generative AI model automates the generation of test text into suitable target illustrations. Specifically, it first extracts key variable parameters from the illustration requirements of the test text and injects these parameters into a structured prompt template. Simultaneously, it combines these parameters with style descriptions of educational illustrations to form a complete generation instruction, resulting in visual prompts. In the diffusion generation stage, the generative AI model not only generates content based on the visual prompts but also adheres to the geometric constraints of ControlNet to ensure the accuracy of the graphic structure of the generated target illustrations. The entire process, supervised by the semantic alignment of the CLIP model, ensures that the output image conforms to the mathematical relationships expressed in the test questions while maintaining the standardization of educational illustrations, ultimately generating target illustrations with accurate variable annotations and harmonious proportions for teaching purposes. The inference process can be described as follows:

[0350]

[0351] in, This represents the original diffusion loss of SD; For controlNet constraint loss; This is the final output image.

[0352] Dynamic parameter adjustment refers to using intelligent algorithms to precisely associate numerical variables in the test text with visual elements, thereby ensuring that the generated target illustrations achieve optimal scientific rigor and pedagogical applicability.

[0353] Dynamic parameter adjustment can be achieved based on a multimodal conditional control system. First, key parameters (such as geometric dimensions and physical quantities) in the test text can be analyzed and dynamically mapped to multiple dimensions of a generative AI model: ControlNet precisely controls the scale of the target illustration (e.g., the radius of a circle strictly matches the labeled number), and the LoRA weight adjuster fine-tunes the generation style (e.g., using different line widths and label densities for circles with 5cm and 10cm radii), while CLIP semantic guidance ensures the reasonable layout of variable labels (e.g., avoiding text obscuring key structures). Within the multimodal conditional system, a built-in teaching knowledge rule base can automatically optimize the presentation of illustrations (e.g., prioritizing the use of unit circles for trigonometric function graphs), and perform multi-level verification of the generated target illustrations (including numerical accuracy checks, scale verification, and teaching standardization assessment), thus ensuring that the final output is both scientifically sound and easy for teaching demonstration. Dynamic parameter adjustment can adapt to the variable generalization needs in educational scenarios, enabling the same type of question to generate visualizations that conform to subject standards under different parameters.

[0354] In addition, when new knowledge points or special question parameter combinations are detected in the question bank, professional illustrations that conform to teaching standards can be automatically generated. These illustrations can be mathematical function graphs with precise variable annotations or chemical experiment flowcharts that demonstrate step by step. The element layout and interaction design can be dynamically adjusted based on the needs of the test questions.

[0355] In this embodiment of the invention, the illustrations generated by the artificial intelligence model can be intelligently optimized and adjusted according to the specific parameters of the test questions: precise numerical correspondence is achieved through variable substitution, the temporal relationship of experimental steps is presented through process decomposition, and interactive elements are added to enhance teaching effectiveness. The adjustment process employs three mechanisms: parameterized template modification, local condition regeneration, and multi-solution optimization. For example, for inclined plane slider problems, comparison diagrams of different inclination angles can be automatically generated and associated with physical formulas. All optimization operations are recorded and analyzed, forming a continuously improving closed-loop learning system to ensure that the illustrations reach their optimal state in terms of scientific validity and pedagogical applicability.

[0356] Based on any of the above embodiments, after step 140, the method further includes:

[0357] Based on the question type of the test question text and the device type of the display device, a layout strategy is determined;

[0358] Based on the layout strategy, the test question text and the target illustration are displayed.

[0359] Specifically, after completing the text-image matching, the successfully matched test text and target illustrations can be laid out. In this embodiment of the invention, the text-image presentation method can be dynamically optimized based on the test text type, the illustration's expressive requirements, and the characteristics of the terminal device. This ensures both the logical coherence of the teaching content and improves visual communication efficiency. This process can be formalized as follows:

[0360]

[0361] in For structured test question text, The target illustration for the test question text. The test data consists of a combination of text and images. To enable the implementation of mixed text and image layouts in the workflow, intelligent line wrapping, dynamic scaling, and interactive element layout can meet teachers' professional requirements for page aesthetics, optimize students' reading flow, significantly improve the usability and dissemination effect of digital teaching resources, and make the visualization of complex knowledge points more in line with cognitive patterns, ultimately maximizing teaching effectiveness.

[0362] Specifically, Figure 6 This is a flowchart illustrating the text and image layout provided by the present invention, such as... Figure 6 As shown, the process can be divided into intelligent layout matching, dynamic content optimization, and multi-format output and compatibility.

[0363] Among them, intelligent layout matching is used to intelligently adapt the text and image layout based on the question type and the device type of the display device, thus obtaining a mixed text and image layout strategy.

[0364] The presentation of images and text can be automatically optimized based on the question type and the type of display device. For example, it can be done through a two-dimensional matrix of question type and device. Automatically select the optimal layout strategy; a layout mapping function can be defined here. as follows:

[0365]

[0366] Where T is the set of question types and D is the equipment type. This indicates the layout strategy.

[0367] In addition, multi-objective optimization algorithms can be used to balance content density and teaching effectiveness. For example, 50% of the page width can be automatically allocated to the derivative graph (such as the f'(x) curve) for mathematical proof problems, and the font size of the formula can be dynamically adjusted to ensure that the core content is not affected when switching between portrait and landscape modes on a tablet.

[0368] Furthermore, complex content flows can be handled through a conflict detection engine. When a long formula or chart in the question text is detected to overlap, horizontal scrolling or pagination can be automatically triggered. The content flow reordering strategy can be defined as follows:

[0369]

[0370] in, The content block can be text, an image, or a formula; ( ) is the overlap detection function.

[0371] Dynamic content optimization refers to the intelligent optimization of display for special content such as long formulas and complex charts, improving the readability of teaching content and user experience.

[0372] For long formulas, an intelligent line-breaking algorithm can dynamically analyze the formula structure and automatically split excessively long formulas while ensuring the integrity of mathematical semantics. Specifically, when a formula length exceeds a threshold, line breaks can be prioritized at relational or higher-order operators, and alignment symbols can be automatically added, thereby ensuring the readability of long formulas when displayed on multiple lines.

[0373] For complex charts, a dynamic rendering technique based on visual saliency can be used. By using a convolutional neural network to identify the core teaching elements in the chart, a multi-level zoom view can be automatically generated, thereby achieving adaptive rendering for complex charts.

[0374] Multi-format output and compatibility refer to support for multiple output formats such as PDF, HTML5, and EPUB, ensuring cross-platform and multi-browser display compatibility and consistency. Specifically, it can display optimized test question text and target illustrations based on layout strategies. Furthermore, it can achieve multi-platform adaptation output of mixed image educational content, including test question text and target illustrations, through an intelligent document conversion engine.

[0375] For example, format conversion can be performed using a conversion framework built on Apache FOP and Pandoc, supporting automatic generation of PDF, HTML5, and EPUB3. PDF output uses the CMYK color gamut and 300dpi resolution; HTML5 integrates MathJax formula rendering and SVG vector illustrations, supporting touch interaction; EPUB implements semantic tag hierarchy. Responsive design is supported, adapting to the display needs of multiple terminals (PC, tablet, mobile phone).

[0376] In this embodiment of the invention, the layout of text and images can be automatically optimized based on the question type and display device. For example, illustrations for multiple-choice questions are typically inlined next to the options, while illustrations for problem-solving questions are placed at key steps in the solution process. The layout engine employs responsive line wrapping or horizontal scrolling for specific content. The final output supports multiple formats and conforms to accessibility standards.

[0377] Based on any of the above embodiments Figure 7 This is the second flowchart illustrating the image-text matching method provided by the present invention, as shown below. Figure 7 As shown, the image-text matching method can be described as follows:

[0378]

[0379] in, , , These consist of the original test data (without illustrations), the intermediate processing steps, and the final output of the text and image data. Based on... Figure 2 The illustrated process, the image-text matching method, may include the following steps:

[0380] First, the raw test question data without illustrations is input into the test question input construction module. Within this module, the raw test question data undergoes standardization preprocessing and structuring, resulting in structured test question text that provides high-quality input for subsequent semantic analysis.

[0381] Secondly, the test question text is input into the test question semantic analysis module. In the test question semantic analysis module, the content of the test question text can be deeply analyzed through natural language processing technology. Then, based on the knowledge point entities and their entity relationships, test question type and test question difficulty in the test question text, the illustration requirements of the test question text are determined.

[0382] Next, the illustration requirement information of the test question text is input into the candidate illustration matching module. In the candidate illustration matching module, the illustration requirement information of the test question text can be matched with the illustration description information of each candidate illustration, thereby obtaining the target illustration that matches the test question text.

[0383] For cases where the target illustration is not among the candidate illustrations, the illustration requirement information from the question text can be input into the dynamic illustration generation module. In this module, the illustration requirement information is converted into visual prompts, and a generative artificial intelligence model is invoked based on these prompts to generate the target illustration.

[0384] After obtaining the target illustration, the test question text and the target illustration can be combined. Figure 1It also includes an adaptive text and image layout module. This module automatically optimizes the text and image layout based on the question type and the display device type, and supports multi-format output, thus balancing aesthetics and functionality to ensure readability and teaching effectiveness across different display devices.

[0385] Figure 8 This is a schematic diagram of the image-text matching system provided by the present invention. Figure 8 Each module shown is applied to Figure 7 The illustrated image-text matching method includes a question input construction module that supports receiving multimodal question data input manually and can perform standardization, cleaning, and structuring of the input question data. The question semantic analysis module can perform semantic analysis on the question text through knowledge point extraction, question type classification, and difficulty classification, and determine the illustration requirements based on the results of the semantic analysis. During knowledge point extraction, entity extraction models can be used to extract knowledge point entities, and entity relationships can be obtained by combining knowledge graphs. The candidate illustration matching module can build an illustration library, annotate corresponding illustration descriptions for each candidate illustration in the library, and calculate the similarity between the illustration requirements of the received question text and the illustration descriptions of each candidate illustration. Additionally, it can match the textual semantics of the question text and the image features of the candidate illustrations based on an attention mechanism. The dynamic illustration generation module can call a generative artificial intelligence (AI) model to dynamically generate target illustrations for the question text, and the content of the target illustrations can be dynamically adjusted based on the parameter settings in the question text. The adaptive graphic layout module can automatically adjust the layout of the target illustrations according to the question type of the test text, including the size and position of the target illustrations, and can adapt to the display of multiple display devices.

[0386] The image-text matching method provided in this invention achieves highly efficient automation of educational image-text synthesis, significantly reducing manual image matching time and improving resource production efficiency. In terms of efficiency, it can speed up the traditional image matching process by more than 10 times, controlling the image matching time for a single question to within 30 seconds, while reducing labor costs by more than 80%, meeting the large-scale application needs of question banks with tens of millions of questions. This method uses deep learning for semantic analysis, breaking through the limitations of traditional keyword matching to ensure that illustrations are highly relevant to the question content. It is innovatively optimized for teaching scenarios, adopting a personalized adaptation mechanism that can dynamically adjust the style and complexity of illustrations according to subject characteristics and question difficulty. This method has strong scalability, supporting both accurate retrieval and matching of illustrations from a preset illustration library and dynamic creation of new illustrations that meet requirements through generative AI, perfectly adapting to diverse educational scenarios such as textbook writing, online question banks, and AR / VR teaching, achieving intelligent, large-scale, and personalized educational content production. Furthermore, through an innovative incremental learning mechanism, the system can continuously absorb new teaching content and graphic styles, maintaining a knowledge point coverage rate of over 95%, providing efficient, accurate, and economical graphic synthesis solutions for scenarios such as textbook writing, online education, and intelligent teaching aids.

[0387] The image-text matching device provided by the present invention is described below. The image-text matching device described below can be referred to in correspondence with the image-text matching method described above.

[0388] Figure 9 This is a schematic diagram of the image-text matching device provided by the present invention. (See diagram below.) Figure 9 As shown, the device includes:

[0389] Unit 910 is used to retrieve the test question text;

[0390] The semantic analysis unit 920 is used to perform knowledge point entity recognition on the test question text to obtain the knowledge point entities in the test question text and the entity relationships of the knowledge point entities, and to perform question type classification and difficulty classification on the test question text to obtain the test question type and test question difficulty of the test question text.

[0391] The requirement determination unit 930 is used to determine the illustration requirement information of the test question text based on the knowledge point entities in the test question text and the entity relationships of the knowledge point entities, the test question type and the test question difficulty of the test question text;

[0392] The demand matching unit 940 is used to determine the target illustration that matches the test question text from the candidate illustrations based on the illustration demand information and the illustration description information of each candidate illustration.

[0393] Based on the above embodiments, the semantic analysis unit is specifically used for:

[0394] Entity recognition is performed on the test question text to obtain the entities in the test question text;

[0395] The entity is mapped to a knowledge graph to obtain the knowledge point entity and the graph node of the knowledge point entity in the knowledge graph;

[0396] Relationship path reasoning is performed on the graph nodes of the knowledge point entity in the knowledge graph to obtain the entity relationship of the knowledge point entity.

[0397] Based on any of the above embodiments, the semantic analysis unit is specifically used for:

[0398] Encode the word features of each word in the test text, wherein the word features include the word vector, position vector and segmentation vector of the word;

[0399] Based on the dependencies between the lexical features of each word in the test text, the contextual features of each word in the test text are determined.

[0400] Entity recognition is performed based on the contextual features of each word in the test text.

[0401] Based on any of the above embodiments, the semantic analysis unit is specifically used for:

[0402] The test texts are classified into macro-level question types, subject-specific categories, and assessment objectives.

[0403] Based on any of the above embodiments, the illustration description information includes at least one of the subject knowledge points, visual features, and teaching attributes of the candidate illustration, and the illustration description information is generated based on a large language model.

[0404] Based on any of the above embodiments, the demand matching unit is specifically used for:

[0405] Based on the similarity between the semantic requirements of the illustration requirement information and the semantic requirements of the illustration description information of each candidate illustration, a target illustration that matches the test question text is determined from the candidate illustrations.

[0406] Based on any of the above embodiments, the demand matching unit is specifically used for:

[0407] Based on the similarity between the requirement semantics of the illustration requirement information and the illustration semantics of the illustration description information of each candidate illustration, and the similarity between the text semantics of the test question text and the image features of each candidate illustration, a target illustration matching the test question text is determined from the candidate illustrations.

[0408] Based on any of the above embodiments, the apparatus further includes an illustration generation unit, used for:

[0409] If no target illustration matching the question text is found among the candidate illustrations, a visual prompt is determined based on the illustration requirement information, and a target illustration matching the question text is generated based on the visual prompt.

[0410] Based on any of the above embodiments, the device further includes a typesetting unit, used for:

[0411] Based on the question type of the test question text and the device type of the display device, a layout strategy is determined;

[0412] Based on the layout strategy, the test question text and the target illustration are displayed.

[0413] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10 As shown, the electronic device may include: a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute a text-image matching method, which includes:

[0414] Obtain the test question text;

[0415] The test question text is subjected to knowledge point entity recognition to obtain the knowledge point entities in the test question text and the entity relationships of the knowledge point entities. The test question text is classified by question type and difficulty to obtain the test question type and test question difficulty of the test question text.

[0416] Based on the knowledge point entities and entity relationships in the test question text, the test question type and difficulty of the test question text, the illustration requirement information of the test question text is determined;

[0417] Based on the illustration requirement information and the illustration description information of each candidate illustration, a target illustration that matches the test question text is determined from the candidate illustrations.

[0418] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0419] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the image-text matching method provided by the above methods, the method comprising:

[0420] Obtain the test question text;

[0421] The test question text is subjected to knowledge point entity recognition to obtain the knowledge point entities in the test question text and the entity relationships of the knowledge point entities. The test question text is classified by question type and difficulty to obtain the test question type and test question difficulty of the test question text.

[0422] Based on the knowledge point entities and entity relationships in the test question text, the test question type and difficulty of the test question text, the illustration requirement information of the test question text is determined;

[0423] Based on the illustration requirement information and the illustration description information of each candidate illustration, a target illustration that matches the test question text is determined from the candidate illustrations.

[0424] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image-text matching method provided by the methods described above, the method comprising:

[0425] Obtain the test question text;

[0426] The test question text is subjected to knowledge point entity recognition to obtain the knowledge point entities in the test question text and the entity relationships of the knowledge point entities. The test question text is classified by question type and difficulty to obtain the test question type and test question difficulty of the test question text.

[0427] Based on the knowledge point entities and entity relationships in the test question text, the test question type and difficulty of the test question text, the illustration requirement information of the test question text is determined;

[0428] Based on the illustration requirement information and the illustration description information of each candidate illustration, a target illustration that matches the test question text is determined from the candidate illustrations.

[0429] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0430] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0431] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for matching images and text, characterized in that, include: Obtain the test question text; The test question text is subjected to knowledge point entity recognition to obtain the knowledge point entities in the test question text and the entity relationships of the knowledge point entities. The test question text is classified by question type and difficulty to obtain the test question type and test question difficulty of the test question text. Based on the knowledge point entities and entity relationships in the test question text, the test question type and difficulty of the test question text, the illustration requirement information of the test question text is determined; Based on the illustration requirement information and the illustration description information of each candidate illustration, a target illustration that matches the test question text is determined from the candidate illustrations.

2. The image-text matching method according to claim 1, characterized in that, The step of performing knowledge point entity recognition on the test question text to obtain the knowledge point entities in the test question text and the entity relationships of the knowledge point entities includes: Entity recognition is performed on the test question text to obtain the entities in the test question text; The entity is mapped to a knowledge graph to obtain the knowledge point entity and the graph node of the knowledge point entity in the knowledge graph; Relationship path reasoning is performed on the graph nodes of the knowledge point entity in the knowledge graph to obtain the entity relationship of the knowledge point entity.

3. The image-text matching method according to claim 2, characterized in that, The entity recognition of the test question text includes: Encode the word features of each word in the test text, wherein the word features include the word vector, position vector and segmentation vector of the word; Based on the dependencies between the lexical features of each word in the test text, the contextual features of each word in the test text are determined. Entity recognition is performed based on the contextual features of each word in the test text.

4. The image-text matching method according to claim 1, characterized in that, The process of classifying the test question text by question type includes: The test texts are classified into macro-level question types, subject-specific categories, and assessment objectives.

5. The image-text matching method according to any one of claims 1 to 4, characterized in that, The illustration description information includes at least one of the subject knowledge points, visual features, and teaching attributes of the candidate illustrations, and the illustration description information is generated based on a large language model.

6. The image-text matching method according to any one of claims 1 to 4, characterized in that, The step of determining the target illustration that matches the test question text from the candidate illustrations based on the illustration requirement information and the illustration description information of each candidate illustration includes: Based on the similarity between the semantic requirements of the illustration requirement information and the semantic requirements of the illustration description information of each candidate illustration, a target illustration that matches the test question text is determined from the candidate illustrations.

7. The image-text matching method according to claim 6, characterized in that, The step of determining the target illustration that matches the test question text from the candidate illustrations based on the similarity between the requirement semantics of the illustration requirement information and the illustration semantics of the illustration description information of each candidate illustration includes: Based on the similarity between the requirement semantics of the illustration requirement information and the illustration semantics of the illustration description information of each candidate illustration, and the similarity between the text semantics of the test question text and the image features of each candidate illustration, a target illustration matching the test question text is determined from the candidate illustrations.

8. The image-text matching method according to any one of claims 1 to 4, characterized in that, Also includes: If no target illustration matching the question text is found among the candidate illustrations, a visual prompt is determined based on the illustration requirement information, and a target illustration matching the question text is generated based on the visual prompt.

9. The image-text matching method according to any one of claims 1 to 4, characterized in that, Also includes: Based on the question type of the test question text and the device type of the display device, a layout strategy is determined; Based on the layout strategy, the test question text and the target illustration are displayed.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the image-text matching method as described in any one of claims 1 to 9.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image-text matching method as described in any one of claims 1 to 9.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image-text matching method as described in any one of claims 1 to 9.

Images