A method for constructing a semantic alignment of an image processing course knowledge base

Abstract

This invention relates to the field of semantic alignment construction of knowledge bases, and more particularly to a method for semantic alignment construction of an image processing course knowledge base. Based on acquired multi-source heterogeneous knowledge data, a standardized transformation process is performed to obtain text semantic features, code execution features, image visual semantic features, and mathematical relation features. These are then uniformly encapsulated to obtain knowledge nodes. Based on these knowledge nodes, text semantic association factors, code processing chain association factors, and visual result association factors are calculated respectively, and the semantic association strength between nodes is generated by fusing them. Knowledge connections are established according to a set association threshold to form an initial knowledge graph. When a new knowledge node is added, the overall impact of the new knowledge node on the existing knowledge structure is calculated to determine whether to trigger relationship re-optimization. This method solves the problems of poor semantic alignment between text, code, images, and formulas, the inability to determine the degree of impact of newly added knowledge on the original knowledge structure, and low knowledge base update efficiency.

Description

Technical Field

[0001] This invention relates to the field of semantic alignment construction of knowledge bases, and more particularly to a method for semantic alignment construction of an image processing course knowledge base. Background Technology

[0002] With the rapid development of computer vision technology, the knowledge content involved in image processing courses is constantly increasing, and course resources are gradually expanding from traditional textbook texts to various data formats such as teaching materials, experimental code, algorithm formulas, teaching videos, and experimental images. Existing methods for constructing image processing course knowledge bases typically store and manage single-type data, mainly establishing relationships between knowledge through keyword matching or text semantic similarity methods. However, image processing course knowledge not only includes conceptual descriptions but also multi-dimensional information such as algorithm execution flows, program implementation processes, mathematical models, and image processing effects. Significant semantic differences exist between different data formats, making it difficult for traditional methods to accurately identify knowledge content with the same technical meaning but different expressions. For example, "edge detection algorithm" in textbooks, "Sobel function call" in experimental code, and "edge enhancement effect" in experimental images belong to text, program, and visual representation forms, respectively. Traditional text-based matching methods struggle to establish effective connections between these three, resulting in incomplete knowledge relationships and insufficient accuracy in semantic connections within the knowledge base. Meanwhile, existing knowledge base construction methods typically employ static maintenance during knowledge updates. When new image processing algorithms, experimental cases, or course chapters are added, the entire knowledge base needs to be traversed again and knowledge relationships recalculated, resulting in significant computational resource consumption. Furthermore, these methods cannot adaptively adjust based on the actual impact of new knowledge on the existing knowledge structure. When new knowledge represents only a local expansion, overall reconstruction leads to substantial unnecessary computations; when new knowledge is related to multiple existing knowledge domains, the fixed structure cannot adjust knowledge relationships in a timely manner, hindering the knowledge base's ability to continuously expand. Therefore, there is an urgent need to propose a semantic alignment construction method for an image processing course knowledge base to address these issues. Summary of the Invention

[0003] This invention provides a semantic alignment construction method for an image processing course knowledge base, which solves the problem that in existing image processing course knowledge bases, it is difficult to establish accurate associations between different forms of course knowledge, resulting in poor semantic alignment between text, code, images, and formulas; at the same time, it is impossible to determine the degree of impact of newly added knowledge on the original knowledge structure, resulting in low knowledge base update efficiency.

[0004] The present invention provides a method for constructing semantic alignment of an image processing course knowledge base, comprising the following steps:

[0005] S1. Acquire multi-source heterogeneous knowledge data, including course textbook text data, teaching courseware data, experimental code data, algorithm formula data, teaching video subtitle data, and experimental image data, and perform standardization transformation to obtain text semantic features, code execution features, image visual semantic features, and mathematical relation features. For the textbook text data and teaching courseware data, text parsing technology is used to extract the raw text data set. This raw text data set undergoes semantic structuring processing to obtain decomposed text knowledge units. Vector encoding is then applied to these decomposed text knowledge units to obtain text semantic features. The teaching video subtitle data is processed using the same semantic structuring and vector encoding methods as the raw text data set. The code results are incorporated into the textual semantic features of the corresponding knowledge nodes. Simultaneously, using video time nodes, a correspondence is established between the content of the teaching video explanations and the pre-processed and post-processed images of the same experimental case, serving as the basis for determining the knowledge content of the image during the visual semantic transformation stage. For algorithm formula data, mathematical relationship features are generated using formula recognition technology. For experimental code data, code execution features are generated through code execution path analysis. Visual semantic transformation is performed on experimental image data to obtain image visual semantic features. The textual semantic features, code execution features, image visual semantic features, and mathematical relationship features are uniformly encapsulated by establishing a cross-modal data grouping index based on course chapter numbers, experimental case identifiers, and video time nodes to obtain knowledge nodes.

[0006] S2. Based on knowledge nodes, a cross-modal semantic coupling alignment algorithm based on processing chain constraints is introduced to calculate the text semantic association factor, code processing chain association factor, and visual result association factor, respectively. Furthermore, the consistency of the mathematical structure of knowledge nodes and the distance between knowledge nodes in the comprehensive semantic space are calculated based on mathematical relationship features to generate the semantic association strength between nodes. The semantic association strength is compared with a set association threshold to establish knowledge connections and form an initial knowledge graph. When a new knowledge node is added, the overall influence of the new knowledge node on the existing knowledge structure is calculated based on the semantic association strength of the new node. The overall influence of the new knowledge node on the existing knowledge structure is compared with a set influence judgment threshold to determine whether to trigger relationship re-optimization.

[0007] Preferably, S2 specifically includes:

[0008] In the cross-modal semantic coupling alignment algorithm based on processing chain constraints, the number of shared concepts between two knowledge nodes and the total number of concepts between the two knowledge nodes are calculated based on the textual semantic features of any two knowledge nodes. The textual semantic association factor is calculated by using the number of shared concepts between the two knowledge nodes as the numerator and the total number of concepts between the two knowledge nodes plus 1 as the denominator.

[0009] Preferably, S2 specifically includes:

[0010] The code execution flow is converted into an execution sequence containing processing actions, i.e., a code execution chain. The text describing the processing actions obtained from the abstract syntax tree parsing is vector-encoded in the same way as the text knowledge units to obtain the text semantic features of the processing actions. Based on the text vectors formed by the text semantic features of two processing actions, the technical semantic similarity between the two processing actions is calculated using cosine similarity. The weight of the processing action is calculated based on the number of data nodes connected by the obtained processing actions. The technical semantic similarity between the two processing actions is multiplied by the weights of the two processing actions to calculate the association contribution of the two processing actions when they have the same technical meaning.

[0011] Preferably, S2 specifically includes:

[0012] Based on the hierarchical position of processing actions in the execution chain, execution position consistency is calculated through exponential decay. The matching contribution value of a single execution action is calculated by multiplying the correlation contribution of two processing actions (assuming technical consistency) with the execution position consistency. Based on the matching contribution value of a single execution action, the best-matching corresponding action and the maximum matching contribution are found. The maximum matching contribution is summed to calculate the cumulative semantic matching contribution of the code execution chain. Based on the processing action weights, the normalization factor of the complexity of the two code execution chains is calculated. The correlation factor of the code processing chain is calculated based on the ratio of the cumulative semantic matching contribution of the code execution chain to the normalization factor of the complexity of the two code execution chains.

[0013] Preferably, S2 specifically includes:

[0014] The system acquires the pre-processed and post-processed images corresponding to two knowledge nodes, and obtains the classification output through a convolutional neural network model. If the image object types corresponding to the two knowledge nodes are different, they are considered to have no direct visual change correlation and are not included in subsequent calculations. If the image object types corresponding to the two knowledge nodes are the same, they are normalized in size, and the images are mapped to the same spatial region for structural division. The grayscale change, edge structure change, and texture change of each region before and after processing are calculated. After normalization, they are added together to obtain the change intensity value of each region, forming the visual change vector of the two knowledge nodes. Based on the change intensity value in the visual change vector, the visual change difference of corresponding regions in the two visual change vectors is compared region by region, and the importance weight of the region is introduced. The visual result correlation factor is calculated by normalizing and weighting the difference.

[0015] Preferably, S2 specifically includes:

[0016] Based on the mathematical structure of mathematical relation features, which includes variable composition relationships, operational hierarchy relationships, and mathematical mapping relationships, the number of shared mathematical structures and the number of mathematical structure differences are obtained by comparing and counting them one by one. The consistency of the mathematical structure of knowledge nodes is obtained by calculating the proportion of the number of shared mathematical structures in the sum of the number of shared mathematical structures and the number of mathematical structure differences. The text semantic features, code execution features, image visual semantic features, and mathematical relation features corresponding to the knowledge nodes are concatenated by connecting the first and last parts. Among them, the code execution features are converted into numerical vectors using the same encoding method of the BERT pre-trained language model as the text knowledge units. The mathematical relation features are numerically encoded according to the dimensions such as the number of variables, the depth of the operational hierarchy, and the number of mapping relationships before participating in the concatenation. The concatenated vector is preprocessed by standardization and dimensionality reduction mapping. The numerical values ​​of each dimension of the dimensionality-reduced feature vector are normalized and used as the coordinates of the knowledge nodes in the knowledge space. Based on the Euclidean distance of the coordinates of the knowledge nodes in the knowledge space, the distance between two knowledge nodes in the comprehensive semantic space is calculated.

[0017] Preferably, S2 specifically includes:

[0018] The formula for calculating semantic association strength is: , in, Representing knowledge nodes With knowledge nodes The final semantic association strength between them; Indicates the semantic association factor of the text; This indicates the chain association factor in the code processing; Indicates the correlation factor of visual results; It is the visual accommodation coefficient; It is the mathematical relationship enhancement coefficient; Indicates the degree of consistency in mathematical structure between two knowledge nodes; This represents the distance between two knowledge nodes in the comprehensive semantic space;

[0019] When either of the two knowledge nodes does not have code execution characteristics Taking a neutral value of 1 indicates that the code dimension provides no additional distinguishing information, neither enhancing nor weakening the association; when any knowledge node does not possess image visual semantic features. The product is 0; when neither knowledge node has a mathematical relationship characteristic. Take 0, Take 1; calculate When missing features are filled in, they are replaced with zero vectors.

[0020] Preferably, S2 specifically includes:

[0021] The semantic association strength is compared with a set association threshold. When the semantic association strength exceeds the association threshold, it is determined that there is a valid semantic mapping relationship between the knowledge nodes, and a knowledge connection is established. When the semantic association strength does not exceed the association threshold, it is determined that there is no valid semantic mapping relationship between the knowledge nodes, and a knowledge connection is not established.

[0022] Preferably, S2 specifically includes:

[0023] Based on the semantic association strength, a knowledge relationship matrix is ​​generated. When a new knowledge node is added to the knowledge base, the semantic association strength between the new node and existing nodes is calculated. The average semantic association strength between the new node and existing nodes is calculated to determine the overall impact of the new knowledge node on the existing knowledge structure.

[0024] Preferably, S2 specifically includes:

[0025] The overall impact of newly added knowledge nodes on the existing knowledge structure is compared with the set impact judgment threshold. When the overall impact of newly added knowledge nodes on the existing knowledge structure is greater than the set impact judgment threshold, it indicates that there is a strong correlation between the newly added knowledge nodes and the existing knowledge system, triggering the knowledge relationship re-optimization process. When the overall impact of newly added knowledge nodes on the existing knowledge structure is less than or equal to the set impact judgment threshold, it indicates that the newly added knowledge nodes only supplement local knowledge content and will not significantly change the existing knowledge structure. The knowledge connection relationship of the newly added knowledge nodes is directly added to the existing knowledge base.

[0026] The beneficial effects of the technical solution of the present invention are:

[0027] 1. By constructing a cross-modal semantic coupling alignment algorithm based on processing chain constraints, knowledge matching no longer relies solely on traditional text similarity. Instead, it comprehensively considers the textual concept relationships, program execution flow relationships, image processing result change relationships, and mathematical model relationships between knowledge nodes. This enables knowledge content with different expressions but the same technical meaning to establish accurate semantic mapping relationships, avoiding knowledge omissions due to name differences and improving the accuracy of knowledge alignment.

[0028] 2. By introducing a code processing chain association factor, the code execution flow is transformed into an execution sequence. The code processing chain association factor is calculated by combining the data flow relationship between processing actions and the hierarchical position in the execution chain. This enables the knowledge base to identify the hidden algorithmic correspondence between different program implementation methods, solving the problem that existing methods cannot understand the actual processing logic of the code and improving the ability to associate theoretical knowledge with experimental implementation knowledge. Attached Figure Description

[0029] Figure 1This is a flowchart of a semantic alignment construction method for an image processing course knowledge base according to the present invention. Detailed Implementation

[0030] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0032] The following description, in conjunction with the accompanying drawings, details a specific scheme for constructing a semantic alignment method for an image processing course knowledge base provided by this invention.

[0033] See attached document Figure 1 The diagram illustrates a flowchart of a semantic alignment construction method for an image processing course knowledge base according to an embodiment of the present invention. The method includes the following steps:

[0034] S1. Acquire multi-source heterogeneous knowledge data, including course textbook text data, teaching courseware data, experimental code data, algorithm formula data, teaching video subtitle data, and experimental image data, and perform standardization transformation to obtain text semantic features, code execution features, image visual semantic features, and mathematical relation features. For the textbook text data and teaching courseware data, text parsing technology is used to extract the raw text data set. This raw text data set undergoes semantic structuring processing to obtain decomposed text knowledge units. Vector encoding is then applied to these decomposed text knowledge units to obtain text semantic features. The teaching video subtitle data is processed using the same semantic structuring and vector encoding methods as the raw text data set. The code results are incorporated into the textual semantic features of the corresponding knowledge nodes. Simultaneously, using video time nodes, a correspondence is established between the content of the teaching video explanations and the pre-processed and post-processed images of the same experimental case, serving as the basis for determining the knowledge content to which the image belongs during the visual semantic transformation stage. For algorithm formula data, mathematical relationship features are generated using formula recognition technology. For experimental code data, code execution features are generated through code execution path analysis. Visual semantic transformation is performed on experimental image data to obtain image visual semantic features. The textual semantic features, code execution features, image visual semantic features, and mathematical relationship features are uniformly encapsulated by establishing a cross-modal data grouping index based on course chapter numbers, experimental case identifiers, and video time nodes to obtain knowledge nodes.

[0035] This application acquires multi-source heterogeneous knowledge data from an image processing course using existing data reading technologies. This multi-source heterogeneous knowledge data includes course textbook text data, teaching materials data, experimental code data, algorithm formula data, teaching video subtitle data, and experimental image data. All data sources for the multi-source heterogeneous knowledge data acquired in this application have been legally authorized.

[0036] Specifically, corresponding data acquisition methods are adopted for different types of course resources: For textbook text data and teaching courseware data, electronic textbook files, PDF files, PPT files and web course documents stored in the course resource server are obtained through the file reading interface, and existing text parsing technologies, such as text semantic parsing technology based on the Bidirectional Encoder Representations from Transformers (BERT) pre-trained language model, are used to extract the text content, title level, chapter number and formula description information to form the original text data set.

[0037] For experimental code data, program files such as Python, MATLAB, and C++ in the course experiment directory are read through the code repository interface or file scanning interface. Existing code parsing technologies, such as abstract syntax tree-based code parsing technology, are used to obtain function call relationships, variable definition information, input parameter types, output parameter types, and program execution path information in the program, forming a raw code data set.

[0038] For subtitle data in teaching videos, the course video files are obtained through a video file reading interface, and existing speech recognition technologies, such as automatic speech recognition technology based on the Transformer structure, are used to convert the teacher's explanations in the video into text descriptions. At the same time, the video time nodes corresponding to each text description are recorded, so that the video explanations can be associated with the course chapters.

[0039] For experimental image data, the system reads the original image files and processed image files from the image processing experiment through the experimental data file interface. These include image data in formats such as JPEG, PNG, and BMP, and simultaneously retrieves the corresponding experimental parameter configuration files. The original image files describe the data state before algorithm processing, the processed image files describe the visual changes after algorithm execution, and the experimental parameter configuration files record the parameter settings during algorithm execution.

[0040] For algorithm formula data, the mathematical formulas in the textbook and the algorithm expressions in the experimental documents are read through the formula parsing interface.

[0041] After obtaining multi-source heterogeneous knowledge data, standardization transformation is performed on different types of knowledge data to convert data from different sources and in different formats into knowledge description data with a unified structure. The specific implementation process is as follows:

[0042] For algorithmic formula data, existing formula recognition technologies, such as Mathpix-based formula recognition technology, are used to convert formulas in image or text form into structured mathematical expressions. This yields a mathematical structure containing variable composition relationships, operational hierarchy relationships, and mathematical mapping relationships, generating mathematical relationship features, which are then denoted as... .

[0043] For the text description information obtained from speech recognition in the subtitle data of teaching videos, the same semantic structuring and vector encoding methods as the text knowledge units are used for processing. The resulting encoding results are incorporated into the text semantic features of the corresponding knowledge nodes. At the same time, using the video time nodes recorded by each segment of text description information, a correspondence is established between the teaching video explanation content and the pre-processed and post-processed images of the same experimental case. This serves as the basis for determining the knowledge content to which the image belongs and realizing cross-modal data grouping in the visual semantic conversion stage.

[0044] The original text data set undergoes semantic structuring, splitting it into a three-level structure of chapters, paragraphs, and sentences. Each split text knowledge unit is assigned a unique text number, and its chapter position, contextual relationship, and dependencies on preceding and following knowledge are recorded. For example, for the following course description: "Convolutional neural networks extract local features from the input image through convolution kernels, reduce feature dimensionality through pooling operations, and finally complete the image classification task," the entire text is not directly treated as a single knowledge node. Instead, it is split into multiple text knowledge units through a semantic decomposition module: "Input object: two-dimensional image matrix; Processing method: convolution kernel sliding calculation; Intermediate process: feature map generation; Optimization process: pooling dimensionality reduction; Target task: image classification." A knowledge flow relationship is established: image matrix → convolution calculation → feature map → pooling → classification result. After the above processing, a text semantic encoding method based on a BERT pre-trained language model is used to vectorize each text knowledge unit, converting text concepts, algorithm descriptions, and technical relationships into text semantic features, which are then denoted as... .

[0045] Subsequently, the experimental code data underwent structured transformation. Since function names in the code may differ from concept names in textbooks, function names were not directly used as semantic basis. Instead, code knowledge descriptions were generated through code execution path analysis. The specific process is as follows: First, abstract syntax tree parsing was performed on experimental code such as Python and MATLAB. For example, `edges=cv2.Canny(image,100,200)` yields the following: "Input variable: image. Input type: 2D pixel matrix. Called function: Canny. Internal processing: gradient calculation, non-maximum suppression, double threshold connection. Output variable: edges". Based on the information obtained after abstract syntax tree parsing, a code knowledge description, i.e., code execution characteristics, was generated. Through this conversion method, even if the textbook uses the description of "edge detection algorithm" while the experimental code uses the "cv2.Canny function" implementation, the two can still establish a correspondence based on the input data, processing flow, and output results.

[0046] Furthermore, visual semantic transformation is performed on the experimental image data. Since the images themselves do not directly describe the meaning of the algorithm, it is necessary to analyze the changes in the images before and after processing. For each experimental case, an image processing procedure is established, with the specific expression as follows: , in, This indicates the image processing procedure; This represents the image before processing. This represents the processed image; Indicates experimental parameters.

[0047] The algorithm utilizes existing image feature extraction methods for calculation. Specifically, it employs the Sobel edge detection operator to obtain edge structure changes, the local binary mode method to obtain texture changes, and the grayscale histogram difference calculation method to obtain grayscale changes. Simultaneously, a pre-trained convolutional neural network model is used to perform deep feature extraction on the processed image, for example, using a ResNet50 structure, including convolutional layers, batch normalization layers, activation function layers, pooling layers, fully connected mapping layers, and a classification output layer. The high-dimensional feature vectors before the classification output layer and the extraction results from the image feature extraction methods are then horizontally concatenated to generate the image visual semantic features. Image visual semantic features are used to describe the actual image changes produced by the algorithm and serve as input for subsequent coordinate calculations of knowledge nodes in the comprehensive semantic space.

[0048] Finally, the text semantic features, code execution features, image visual semantic features, and mathematical relationship features obtained after standardized transformation are uniformly encapsulated. A cross-modal data grouping index is established based on course chapter numbers, experimental case identifiers, and video time nodes. Experimental codes, mathematical relationships, experimental parameter configuration files, and images before and after processing within the same experimental case directory are grouped into the same knowledge node. Textual knowledge units are grouped into corresponding knowledge nodes according to their chapter position, subtitle time node, and reference relationship to the experimental case. Each knowledge node also retains an index of its corresponding image processing pair, used for retrieving images before and after processing when calculating visual result correlation factors. Standardized Knowledge Nodes The specific expression is: , in, Indicates the first One knowledge node; Indicates the first Textual semantic features of each knowledge node; Indicates the first Code execution characteristics of each knowledge node; Indicates the first Image visual semantic features of each knowledge node; Indicates the first The mathematical relationship characteristics of each knowledge node. Each knowledge node represents an independently describable technical object, algorithm process, mathematical model, program implementation process, or experimental result within the image processing course knowledge system. Each knowledge node corresponds to at least one form of course knowledge expression, and a unified data description structure is established through textual semantic features, code execution features, image visual semantic features, and mathematical relationship features. Ultimately, all knowledge nodes form a unified knowledge representation space. , This represents the total number of knowledge nodes.

[0049] S2. Based on knowledge nodes, a cross-modal semantic coupling alignment algorithm based on processing chain constraints is introduced to calculate the text semantic association factor, code processing chain association factor, and visual result association factor, respectively. Furthermore, the consistency of the mathematical structure of knowledge nodes and the distance between knowledge nodes in the comprehensive semantic space are calculated based on mathematical relationship features to generate the semantic association strength between nodes. The semantic association strength is compared with a set association threshold to establish knowledge connections and form an initial knowledge graph. When a new knowledge node is added, the overall influence of the new knowledge node on the existing knowledge structure is calculated based on the semantic association strength of the new node. The overall influence of the new knowledge node on the existing knowledge structure is compared with a set influence judgment threshold to determine whether to trigger relationship re-optimization.

[0050] After completing the above data collection and standardization transformation, the heterogeneous course knowledge, originally existing in the form of text, code, images, and mathematical formulas, is mapped to a unified knowledge representation space. This approach aligns data structures across different knowledge representation formats, facilitating knowledge node fusion. To address the issue of incompatibility due to differing knowledge representation methods from different sources, a cross-modal semantic coupling alignment algorithm based on processing chain constraints is introduced. Unlike traditional methods that calculate text vector distance, this algorithm does not directly determine name similarity but instead assesses: whether the data input is identical; whether similar processing steps have been performed; and whether the results are of the same type. The specific implementation process is as follows:

[0051] First, calculate the text semantic association factor. For any two knowledge nodes... and Extracting semantic features from text and Further calculate the proportion of common textual knowledge units. ,in, It represents the proportion of shared textual knowledge units, i.e., the textual semantic association factor, which is used to reflect the degree of textual association. It is the first The textual semantic features of the knowledge node represent the first knowledge node. The set consisting of the concepts contained in a knowledge node and their encoding vectors; Indicates the first Textual semantic features of each knowledge node; This indicates the number of shared concepts between two knowledge nodes. Each text knowledge unit obtained through semantic structuring is used as a concept. When the cosine similarity of the encoding vectors of two concepts is greater than a preset concept matching threshold (e.g., 0.85), they are determined to be shared concepts. This represents the total number of concepts in two knowledge nodes.

[0052] However, since textual similarity does not necessarily imply technological similarity, textual correlation cannot be directly used for knowledge linking. Furthermore, we calculate the code processing chain correlation factor, transforming the code execution flow into an execution sequence, i.e., a code execution chain, with the specific expression as follows: , in, , They represent the first The execution sequence of the knowledge node, the first The execution sequence of each knowledge node; , They represent the first In the execution sequence of the nth knowledge node The first processing action, the first In the execution sequence of the nth knowledge node One processing action; , The first The knowledge node and the first The index of the processing action in the execution sequence of each knowledge node; , They represent the first The knowledge node and the first The total number of actions processed in the execution sequence of each knowledge node.

[0053] Then, the correspondence between the two execution sequences is calculated, specifically as follows: , in, This indicates the degree of association in the code execution chain, i.e., the association factor in the code processing chain; Indicates the first In the execution sequence of the nth knowledge node The first processing action and the first In the execution sequence of the nth knowledge node The text describes the semantic similarity of processing actions, and then performs vector encoding on the text describing the processing actions obtained from the abstract syntax tree parsing using the same text semantic encoding method as the text knowledge units. This yields the text semantic features of the processing actions. Text vectors composed of text semantic features and The degree of technical semantic similarity is obtained by calculating the cosine similarity of text vectors composed of text semantic features; Indicates the first In the execution sequence of the nth knowledge node The weight of each processing action is calculated based on the number of data nodes connected to that action. , Indicates the first In the execution sequence of the nth knowledge node The number of data nodes connected to each processing action. A data node is a node in the code execution chain that represents the data object read, generated, or passed by the processing action, including input variables, intermediate variables, and output variables, and is obtained by counting. Indicates the first In the execution sequence of the nth knowledge node Each processing action weight, same The calculation method is the same; Indicates the first In the execution sequence of the knowledge node, the first... The hierarchical position of each processing action in the execution chain is essentially the position of the first... The position number of each processing action in the code execution chain can also be understood as which action it is; Indicates the first In the execution sequence of the knowledge node, the first... The hierarchical position of each processing action in the execution chain; Indicates the first The set of indices of the actions processed in the execution sequence of each knowledge node; Indicates the importance of an action; This indicates the correlation contribution between two processing actions when they are technically equivalent. It is the cumulative contribution of semantic matching in the code execution chain, used to calculate the cumulative matching strength between the program execution flows corresponding to two knowledge nodes; It refers to the consistency of execution position, which indicates the continuity of two actions in the execution chain. Indicates the first In the execution sequence of the nth knowledge node The first processing action and the first In the execution sequence of the nth knowledge node The contribution value of matching single execution actions among processing actions; It is the maximum matching contribution, representing the maximum matching contribution for the i-th... In the execution sequence of the nth knowledge node The processing action, in the first The best matching action is found among all the processing actions in the execution sequence of each knowledge node, and the maximum matching contribution is taken as the best correspondence between the two execution chains for that action. This represents the normalization factor of the complexity of the two code execution chains themselves.

[0054] Furthermore, the visual result association factor is calculated. Specifically, the pre-processed and post-processed images corresponding to the two knowledge nodes are obtained respectively. The classification output result is obtained through the pre-trained convolutional neural network model mentioned above. It is determined whether the images corresponding to the two knowledge nodes belong to the same type of processing object. If the two image objects are of different types, it is considered that there is no direct visual change association between them, and they are not included in the subsequent calculation. If the two image objects are of the same type, the two images are further normalized in size using an interpolation scaling method, such as bilinear interpolation, based on uniform grid partitioning. Images of different resolutions are mapped to the same spatial region partitioning structure. Only the image size is normalized without changing the original numerical range of pixel grayscale values. For example, each image is divided into the same number of... The region, and ensure the first Each region has the same spatial semantic location in both images. Subsequently, the visual change features of each region before and after processing are calculated, including gray-level changes, edge structure changes, and texture changes calculated using the aforementioned Sobel operator, local binary mode, and gray-level histogram difference method. The gray-level changes, edge structure changes, and texture changes are normalized using their maximum and minimum values ​​and then summed to obtain the change intensity value for each region, forming the first... Visual change vector of each knowledge node and the Visual change vector of each knowledge node After obtaining two change vectors with the same number of regions and corresponding spatial locations, the visual change differences of corresponding regions in the two visual change vectors are compared region by region, and importance weights are assigned to regions according to their importance to the semantic expression of the image processing results. Regions with significant edge changes are given higher weights, while regions with flat backgrounds are given lower weights. Finally, a normalized weighted difference is used to calculate the consistency of changes between the two image processing results. The calculation process is as follows: , in, It represents the degree of consistency in the changes of the image processing results corresponding to two knowledge nodes, i.e., the visual result association factor; Indicates the first The knowledge node corresponds to the image of the first knowledge node. The intensity of change in each region; Indicates the first The knowledge node corresponds to the image of the first knowledge node. The intensity of change in each region; This represents the number of image regions after size normalization, which is the number of fixed regions shared by both images; Indicates the first The importance weight of each region is calculated as the ratio of the sum of the region's edge density, texture complexity, and the target region's proportion to the sum of all regions, satisfying the following conditions: And all regional weights satisfy ; This represents a numerical stability factor to prevent the denominator from being zero; the reference range is... to ; Represents the two visual change vectors at the th... Differences in visual changes produced by different regions; This represents the overall deviation in change between two visual change vectors; This represents the total visual change generated by two visual change vectors; It indicates the proportion of the change.

[0055] Furthermore, based on the multidimensional association features composed of the aforementioned text semantic association factors, code processing chain association factors, and visual result association factors, the latent semantic correspondence between knowledge nodes from different sources is calculated. This establishes a semantic mapping relationship between knowledge nodes with the same technical meaning but different forms of expression, thereby achieving cross-modal semantic alignment of image processing course knowledge. The semantic association strength is calculated, and the specific expression is as follows: , in, Representing knowledge nodes With knowledge nodes The final semantic association strength between them; This indicates the degree of consistency in cross-modal technical content between knowledge nodes; Indicates the consistency enhancement factor of mathematical mechanism; This indicates the coupling relationship between textual conceptual associations and code execution associations; This is the visual accommodation coefficient, used to avoid excessive influence from visual information. It is determined using a five-fold cross-validation method, and the reference value range is [value range missing]. For example, 0.3; It is the mathematical relationship enhancement coefficient, determined using the five-fold cross-validation method, with a reference value range of [value missing]. For example, 0.2; This indicates the degree of consistency in mathematical structure between two knowledge nodes. ,in, This indicates the total number of mathematical structures shared by the two knowledge nodes. This represents the number of differences in mathematical structures; mathematical structures are corresponding... and The relationships between variables, the hierarchical relationships of operations, and the mathematical mapping relationships are compared and counted one by one to obtain the number of identical mathematical structures. and the number of different mathematical structures ; The distance between two knowledge nodes in the comprehensive semantic space is calculated using Euclidean distance based on the coordinates of the knowledge nodes in the knowledge space. The coordinates of the knowledge nodes in the knowledge space are obtained by concatenating the textual semantic features, code execution features, image visual semantic features, and mathematical relation features corresponding to the knowledge nodes according to the concatenation rules of head-to-tail connection. Among them, the code execution features are converted into numerical vectors using the same encoding method of the BERT pre-trained language model as the text knowledge units. The mathematical relation features are numerically encoded according to the dimensions such as the number of variables, the depth of the operation level, and the number of mapping relationships before participating in the concatenation. After the concatenated vector is standardized and preprocessed using Z-score standardization technology, it is input into an existing feature dimensionality reduction algorithm, such as principal component analysis, for dimensionality reduction mapping. The dimensionality-reduced feature vector retains 95% of the variance. After the values ​​of each dimension of the dimensionality-reduced feature vector are normalized to maximum and minimum, it is used as the coordinates of the knowledge node in the knowledge space.

[0056] When either of the two knowledge nodes does not have code execution characteristics, let Taking a neutral value of 1 indicates that the code dimension does not provide additional distinguishing information, neither enhancing nor weakening the association. and The product of these factors degenerates into the text semantic association factor itself; when any knowledge node does not possess image visual semantic features, let and The product is set to 0; when neither knowledge node has a mathematical relation characteristic, let Set the value to 0, and the mathematical mechanism consistency enhancement factor to 1; calculate When missing features are filled in, they are replaced with zero vectors.

[0057] Furthermore, the calculated semantic association strength is used as the evaluation result of the semantic alignment of knowledge nodes. The semantic association strength is compared with a set association threshold. When the semantic association strength exceeds the association threshold, the knowledge node is determined. and There are valid semantic mapping relationships between them, establishing knowledge connections: It generates an initial knowledge graph; when the semantic association strength does not exceed the association threshold, it determines the knowledge nodes. and There is no mapping relationship between them, and no knowledge connection is established. The association threshold is determined by arranging all semantic association strengths from smallest to largest and selecting the maximum value corresponding to the top 80% of semantic association strength values ​​as the association threshold.

[0058] However, as course content continues to increase, such as adding new content on Transformer and diffusion models, the original knowledge relationships may change. Therefore, a dynamic relationship correction mechanism based on knowledge flow topology feedback is introduced. The specific implementation process is as follows:

[0059] Read the semantic association strength obtained above And generate a knowledge relationship matrix. ;

[0060] When a new knowledge node is added to the knowledge base, the degree of association between the new node and existing knowledge nodes is calculated to determine whether the new knowledge has changed the existing semantic mapping relationship. Semantic alignment calculations are then re-executed when necessary, achieving dynamic maintenance of the semantic relationships in the knowledge base. That is, when a new knowledge node is added... When adding a node, first calculate the semantic association strength between the new node and existing nodes. This forms a new relation matrix. Then, calculate the overall impact of the newly added knowledge nodes on the existing knowledge structure. The specific expression is: , in, This indicates the overall impact of newly added knowledge nodes on the existing knowledge structure. Indicates the addition of a new knowledge node With the The semantic association strength between existing knowledge nodes; This represents the total number of existing knowledge nodes in the knowledge base before a new knowledge node is added.

[0061] When calculated When the impact threshold is exceeded, it indicates a strong correlation between the new knowledge node and the existing knowledge system, potentially altering the original knowledge organization structure. This triggers a knowledge relationship re-optimization process, recalculating semantic relationships between knowledge nodes whose semantic association with the new knowledge node exceeds the threshold and their first-order neighbors, updating the connections between knowledge nodes. When the impact threshold is less than or equal to the set threshold, it indicates that the new knowledge node only supplements local knowledge content and does not significantly change the existing knowledge structure. The new knowledge node is then directly added to the existing knowledge base according to the calculated knowledge connections, thus completing the dynamic expansion of the knowledge base. This method avoids indiscriminate reconstruction of the entire knowledge graph, reduces the computational load during knowledge updates, and ensures that new course content can adaptively adjust according to its actual impact on the original knowledge system, achieving continuous semantic alignment construction of the image processing course knowledge base. During the knowledge base construction phase, each existing knowledge node is treated as a new knowledge node, and the overall impact of the new knowledge node on the existing knowledge structure is calculated to form an impact sample set. After sorting, the value corresponding to the 80th percentile is selected as the impact threshold. This sample set is continuously updated as new events accumulate during the knowledge base operation.

[0062] In summary, a method for constructing semantic alignment of an image processing course knowledge base has been completed.

[0063] The order of the embodiments is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0064] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0065] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for constructing semantic alignment in an image processing course knowledge base, characterized in that, Includes the following steps: S1. Acquire multi-source heterogeneous knowledge data, including course textbook text data, teaching courseware data, experimental code data, algorithm formula data, teaching video subtitle data, and experimental image data, and perform standardized transformation processing to obtain text semantic features, code execution features, image visual semantic features, and mathematical relationship features. For textbook text data and teaching courseware data, text parsing technology is used to extract the raw text data set. The raw text data set is then semantically structured to obtain decomposed text knowledge units. Vector encoding is performed on the decomposed text knowledge units to obtain text semantic features. For teaching video subtitle data, the same semantic structured processing and vector encoding method as the raw text data set is used. The resulting encoding results are incorporated into the text semantic features of the corresponding knowledge nodes. At the same time, using video time nodes, a correspondence is established between the teaching video explanation content and the pre-processed and post-processed images of the same experimental case, serving as the basis for determining the knowledge content to which the image belongs in the visual semantic transformation stage. For algorithm formula data, formula recognition technology is used to generate mathematical relationship features. Code execution characteristics are generated from experimental code data through code execution path analysis; Visual semantic transformation is performed on experimental image data to obtain image visual semantic features; text semantic features, code execution features, image visual semantic features, and mathematical relationship features are uniformly encapsulated by establishing a cross-modal data grouping index based on course chapter number, experimental case identifier, and video time node to obtain knowledge nodes. S2. Based on knowledge nodes, a cross-modal semantic coupling alignment algorithm based on processing chain constraints is introduced to calculate the text semantic association factor, code processing chain association factor, and visual result association factor, respectively. Furthermore, the consistency of the mathematical structure of knowledge nodes and the distance between knowledge nodes in the comprehensive semantic space are calculated based on mathematical relationship features to generate the semantic association strength between nodes. The semantic association strength is compared with a set association threshold to establish knowledge connections and form an initial knowledge graph. When a new knowledge node is added, the overall influence of the new knowledge node on the existing knowledge structure is calculated based on the semantic association strength of the new node. The overall influence of the new knowledge node on the existing knowledge structure is compared with a set influence judgment threshold to determine whether to trigger relationship re-optimization.

2. The method for constructing semantic alignment of an image processing course knowledge base according to claim 1, characterized in that, S2 specifically includes: In the cross-modal semantic coupling alignment algorithm based on processing chain constraints, the number of shared concepts between two knowledge nodes and the total number of concepts between the two knowledge nodes are calculated based on the textual semantic features of any two knowledge nodes. The textual semantic association factor is calculated by using the number of shared concepts between the two knowledge nodes as the numerator and the total number of concepts between the two knowledge nodes plus 1 as the denominator.

3. The method for constructing semantic alignment of an image processing course knowledge base according to claim 1, characterized in that, S2 specifically includes: The code execution flow is converted into an execution sequence containing processing actions, i.e., a code execution chain. The text describing the processing actions obtained from the abstract syntax tree parsing is vector-encoded in the same way as the text knowledge units to obtain the text semantic features of the processing actions. Based on the text vectors formed by the text semantic features of two processing actions, the technical semantic similarity between the two processing actions is calculated using cosine similarity. The weight of the processing action is calculated based on the number of data nodes connected by the obtained processing actions. The technical semantic similarity between the two processing actions is multiplied by the weights of the two processing actions to calculate the association contribution of the two processing actions when they have the same technical meaning.

4. The method for constructing semantic alignment of an image processing course knowledge base according to claim 3, characterized in that, S2 specifically includes: Based on the hierarchical position of processing actions in the execution chain, execution position consistency is calculated through exponential decay. The matching contribution value of a single execution action is calculated by multiplying the correlation contribution of two processing actions (assuming technical consistency) with the execution position consistency. Based on the matching contribution value of a single execution action, the best-matching corresponding action and the maximum matching contribution are found. The maximum matching contribution is summed to calculate the cumulative semantic matching contribution of the code execution chain. Based on the processing action weights, the normalization factor of the complexity of the two code execution chains is calculated. The correlation factor of the code processing chain is calculated based on the ratio of the cumulative semantic matching contribution of the code execution chain to the normalization factor of the complexity of the two code execution chains.

5. The method for constructing semantic alignment of an image processing course knowledge base according to claim 1, characterized in that, S2 specifically includes: The system acquires the pre-processed and post-processed images corresponding to two knowledge nodes, and obtains the classification output through a convolutional neural network model. If the image object types corresponding to the two knowledge nodes are different, they are considered to have no direct visual change correlation and are not included in subsequent calculations. If the image object types corresponding to the two knowledge nodes are the same, they are normalized in size, and the images are mapped to the same spatial region for structural division. The grayscale change, edge structure change, and texture change of each region before and after processing are calculated. After normalization, they are added together to obtain the change intensity value of each region, forming the visual change vector of the two knowledge nodes. Based on the change intensity value in the visual change vector, the visual change difference of corresponding regions in the two visual change vectors is compared region by region, and the importance weight of the region is introduced. The visual result correlation factor is calculated by normalizing and weighting the difference.

6. The method for constructing semantic alignment of an image processing course knowledge base according to claim 5, characterized in that, S2 specifically includes: Based on the mathematical structure of mathematical relation features, which includes variable composition relationships, operational hierarchy relationships, and mathematical mapping relationships, the number of shared mathematical structures and the number of mathematical structure differences are obtained by comparing and counting them one by one. The consistency of the mathematical structure of knowledge nodes is obtained by calculating the proportion of the number of shared mathematical structures in the sum of the number of shared mathematical structures and the number of mathematical structure differences. The text semantic features, code execution features, image visual semantic features, and mathematical relation features corresponding to the knowledge nodes are concatenated by connecting the first and last parts. Among them, the code execution features are converted into numerical vectors using the same encoding method of the BERT pre-trained language model as the text knowledge units. The mathematical relation features are numerically encoded according to the dimensions such as the number of variables, the depth of the operational hierarchy, and the number of mapping relationships before participating in the concatenation. The concatenated vector is preprocessed by standardization and dimensionality reduction mapping. The numerical values ​​of each dimension of the dimensionality-reduced feature vector are normalized and used as the coordinates of the knowledge nodes in the knowledge space. Based on the Euclidean distance of the coordinates of the knowledge nodes in the knowledge space, the distance between two knowledge nodes in the comprehensive semantic space is calculated.

7. The method for constructing semantic alignment of an image processing course knowledge base according to claim 6, characterized in that, S2 specifically includes: The formula for calculating semantic association strength is: in, Representing knowledge nodes With knowledge nodes The final semantic association strength between them; Indicates the semantic association factor of the text; This indicates the chain association factor in the code processing; Indicates the correlation factor of visual results; It is the visual accommodation coefficient; It is the mathematical relationship enhancement coefficient; Indicates the degree of consistency in mathematical structure between two knowledge nodes; This represents the distance between two knowledge nodes in the comprehensive semantic space; When either of the two knowledge nodes does not have code execution characteristics Taking a neutral value of 1 indicates that the code dimension provides no additional distinguishing information, neither enhancing nor weakening the association; when any knowledge node does not possess image visual semantic features. The product is 0; when neither knowledge node has a mathematical relationship characteristic. Take 0, Set to 1; calculate When missing features are filled in, they are replaced with zero vectors.

8. The method for constructing semantic alignment of an image processing course knowledge base according to claim 7, characterized in that, S2 specifically includes: The semantic association strength is compared with a set association threshold. When the semantic association strength exceeds the association threshold, it is determined that there is a valid semantic mapping relationship between the knowledge nodes, and a knowledge connection is established. When the semantic association strength does not exceed the association threshold, it is determined that there is no valid semantic mapping relationship between the knowledge nodes, and a knowledge connection is not established.

9. The method for constructing semantic alignment of an image processing course knowledge base according to claim 8, characterized in that, S2 specifically includes: Based on the semantic association strength, a knowledge relationship matrix is ​​generated. When a new knowledge node is added to the knowledge base, the semantic association strength between the new node and existing nodes is calculated. The average semantic association strength between the new node and existing nodes is calculated to determine the overall impact of the new knowledge node on the existing knowledge structure.

10. A method for constructing semantic alignment of an image processing course knowledge base according to claim 9, characterized in that, S2 specifically includes: The overall impact of newly added knowledge nodes on the existing knowledge structure is compared with the set impact judgment threshold. When the overall impact of newly added knowledge nodes on the existing knowledge structure is greater than the set impact judgment threshold, it indicates that there is a strong correlation between the newly added knowledge nodes and the existing knowledge system, triggering the knowledge relationship re-optimization process. When the overall impact of newly added knowledge nodes on the existing knowledge structure is less than or equal to the set impact judgment threshold, it indicates that the newly added knowledge nodes only supplement local knowledge content and will not significantly change the existing knowledge structure. The knowledge connection relationship of the newly added knowledge nodes is directly added to the existing knowledge base.

Images