Universal intelligent splitting method for knowledge points in education field

Abstract

The invention provides a universal intelligent splitting method for knowledge points in the education field. The universal intelligent splitting method comprises the following steps: acquiring videos, audios and document resources related to courses; obtaining a triple representation of video knowledge and a description thereof, a triple representation of audio knowledge and a triple representation of document knowledge and a description thereof from a video, an audio and a document; performing entity disambiguation and anaphora disambiguation on the triad representation and description of the video knowledge, the triad representation and description of the audio knowledge and the triad representation and description of the document knowledge to obtain disambiguated knowledge triads and description of the disambiguated knowledge triads; knowledge point fragments in the corresponding modal resources are intercepted through description of the knowledge triad, and the knowledge point fragments in the multi-modal resources are associated according to entity disambiguation and anaphora disambiguation conditions of entities in the knowledge triad. According to the invention, through multi-modal identification, the accuracy, the precision rate and the recall rate of knowledge point identification are improved, and the proportion of human intervention of teachers is reduced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent video knowledge point splitting, and in particular to a general method for intelligent knowledge point splitting in the field of education. Background Art

[0002] With the rapid development of artificial intelligence technology, its application in the field of education is becoming increasingly widespread. It can realize automatic recording of classroom content and intelligent extraction of knowledge points. Through video recognition AI capabilities, it supports innovative applications such as knowledge point extraction and video slicing, providing students with more personalized learning resources, and can also help teachers better reflect on teaching and professional development. Existing AI video intelligent analysis technology uses deep learning and computer vision technology to automatically extract useful information from videos and achieve efficient and accurate understanding and application of video content. It can be used in the field of education to help teachers and students extract key knowledge points from videos. However, in complex scenarios, especially when multiple objects and behaviors are involved, the accuracy of recognition may still be insufficient. This may lead to the omission or incorrect extraction of key information, thereby affecting the teaching effect, that is, there is a problem of technical accuracy. If the training data of the AI ​​system is biased, it may cause the algorithm to produce unfair results when analyzing. This bias not only affects the accuracy of the analysis, but also may cause unfair results in educational decision-making, that is, algorithmic bias. In particular, deep learning models are often regarded as "black boxes" and their decision-making process is not easy to understand. This lack of transparency may lead to reduced trust in AI systems among educators and students, affecting their use in teaching, namely lack of transparency and explainability. Summary of the invention

[0003] In response to the problems existing in the existing technology, a general method for intelligent splitting of knowledge points in the field of education is provided. Through multimodal recognition and combined with knowledge graphs, knowledge extraction is directly performed to solve the problem of low accuracy of traditional solutions.

[0004] The present invention proposes a general method for intelligently splitting knowledge points in the field of education, comprising:

[0005] Get video, audio and document resources related to the course;

[0006] Acquire triple representation and description of video knowledge, triple representation and description of audio knowledge, and triple representation and description of document knowledge from video, audio, and document;

[0007] Perform entity disambiguation and reference disambiguation on the triple representation and description of video knowledge, the triple representation and description of audio knowledge, and the triple representation and description of document knowledge to obtain disambiguated knowledge triples and descriptions;

[0008] The knowledge point fragments in the corresponding modal resources are intercepted through the description of the knowledge triples, and the knowledge point fragments in the multimodal resources are associated according to the entity disambiguation and reference disambiguation of the entities in the knowledge triples.

[0009] As a preferred solution, it also includes, after obtaining the disambiguated knowledge triples and their descriptions, using the existing teaching knowledge graph to standardize the knowledge triples and their descriptions.

[0010] As a preferred solution, the triple representation of the video knowledge and the acquisition process of its description specifically include:

[0011] Identify object information in the teaching scene from video resources, including projector information and computer desktop information;

[0012] Segment the video according to the projector information and the computer desktop information to obtain projector segment group segments and computer desktop segment group segments;

[0013] Identify text boxes and their corresponding spatial structure coordinates in the projector segment group segment and the computer desktop segment group segment;

[0014] Scoring the text boxes and determining key text fragments with scores above a preset threshold;

[0015] Use NLP technology to analyze key text fragments, obtain their semantic information and filter out meaningless text to obtain the filtered video information text group;

[0016] Various entity data are obtained from the video information text group, and the relationship between each entity is analyzed to construct a triple representation of video knowledge and its description.

[0017] As a preferred solution, the triple representation of the audio knowledge and the process of obtaining its description specifically include:

[0018] Convert audio resources into text information, and use NLP technology to correct the text information to obtain a corrected audio information text group;

[0019] Various entity data are obtained from the audio information text group, and the relationship between each entity is analyzed to construct a triple representation of audio knowledge and its description.

[0020] As a preferred solution, the triple representation of the document knowledge and the acquisition process of its description specifically include:

[0021] For plain text document resources, extract text information through document tools;

[0022] For image version document resources, we extract the text in the image through OCR, perform layout analysis to determine the reading information of the text, and then use NLP technology to analyze the semantics and merge them into a complete text information;

[0023] For document resources that contain both text and images, they are first converted into image-version document resources, and then the text information is extracted using the processing method for image-version document resources.

[0024] Merge text information from plain text document resources, image version document resources, and mixed image and text document resources into a document information text group;

[0025] Various entity data are obtained from the document information text group, and the relationship between each entity is analyzed to construct a triple representation and description of the document knowledge.

[0026] As a preferred solution, the entity disambiguation includes:

[0027] Merge the triple representation and description of video knowledge, the triple representation and description of audio knowledge, and the triple representation and description of document knowledge into a total triple set, and obtain the corresponding entity set;

[0028] Use the pre-trained embedded model to vectorize each triple and its description in the total triple set to obtain the corresponding vector value;

[0029] Divide the entities in the entity combination into different types of sub-entity sets;

[0030] In each type of sub-entity set, calculate the similarity between each triple and the vector value corresponding to its description;

[0031] Two sets of vector values ​​with similarity higher than a preset threshold are considered similar and the two entities are merged; two sets of vector values ​​with similarity lower than a preset threshold are considered dissimilar;

[0032] After completing the vector value similarity calculation between all triples, the set of triples and the corresponding entity set after entity disambiguation are obtained.

[0033] As a preferred solution, the reference disambiguation includes:

[0034] Store the pronoun entities in the entity type as a pronoun set;

[0035] Use the end-to-end model to extract an entity from the pronoun set and entity set respectively, input its corresponding triple and description into the model, and the model will directly output a reference probability;

[0036] When the reference probability is greater than a preset reference probability threshold, the pronoun is regarded as a reference to the corresponding entity, and the pronoun entity in the triplet where the pronoun is located is replaced with the corresponding entity;

[0037] After completing the probability calculation between all triples and pronouns, the set of triples and the corresponding entity set after reference disambiguation are obtained.

[0038] As a preferred solution, the method of extracting knowledge point fragments in the corresponding modal resource through the description of knowledge triples specifically includes:

[0039] Directly extract knowledge point fragments in the corresponding modal resource according to the description of the knowledge triples, wherein the description of the knowledge triples includes the expression of the triples in the original text and its adjacent context; wherein, if the description of the knowledge triples has both video text context and audio text context, determine whether they are from the same audio video resource, if not, extract the video fragment from the corresponding video start frame to the end frame when the video text context is identified as the knowledge point fragment of the original video, and extract the audio fragment from the corresponding audio start time code to the end time code when the audio text context is identified as the audio knowledge point fragment; if yes, only extract the corresponding video fragment from the video start frame to the end frame as the knowledge point fragment of the original video.

[0040] As a preferred solution, the method of associating knowledge point fragments in multimodal resources according to entity disambiguation and coreference disambiguation of entities in knowledge triples specifically includes:

[0041] In the process of entity disambiguation and reference disambiguation, if the entity of the knowledge triple has a corresponding disambiguated entity, the entity and the modal resource where the corresponding disambiguated entity is located are associated resources, and the corresponding knowledge point fragments are associated by using the entity as the association point.

[0042] As a preferred solution, the use of the existing teaching knowledge graph to standardize the knowledge triples and their descriptions specifically includes:

[0043] Convert the existing teaching knowledge graph into a graph triple representation;

[0044] Use the embedded model to vectorize the graph triples and disambiguated knowledge triples and calculate the similarity;

[0045] According to the similarity calculation results, find the knowledge points of the knowledge triples in the teaching knowledge graph;

[0046] The big model tool is then used to compare the knowledge triples and their descriptions with the descriptions of the knowledge points in the teaching knowledge graph to determine whether there are any errors. If there are deviations, the knowledge triples and their descriptions are standardized to finally obtain the standardized knowledge triples and their descriptions.

[0047] Compared with the prior art, the beneficial effects of adopting the above technical solution are as follows: the present invention improves the accuracy, precision and recall rate of knowledge point recognition by combining resources of different modalities, increases the proportion of automation, reduces the proportion of teacher intervention, and achieves better recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of the general method for intelligent splitting of knowledge points in the field of education proposed by the present invention. DETAILED DESCRIPTION

[0049] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar modules or modules with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be construed as limitations on the present application. On the contrary, the embodiments of the present application include all changes, modifications and equivalents that fall within the spirit and connotation of the appended claims.

[0050] Existing technologies can only extract entity information involved in video images through entity recognition in deep learning. Such recognized entities are not necessarily the knowledge points required for teaching, and teachers still need to manually screen them later. In addition, entity recognition is limited by the data samples pre-trained by the deep learning model. There is a lack of data for some professional vocabulary or emerging knowledge points in the industry, which makes it impossible to recognize the corresponding entities, and ultimately miss knowledge points.

[0051] Based on this, the embodiment of the present invention proposes a general method for intelligently splitting knowledge points in the field of education, which solves the above problems by directly extracting knowledge through multi-modal recognition combined with knowledge graphs. The specific solution is as follows:

[0052] Step 1: Obtain video, audio and document resources related to the course.

[0053] In this embodiment, the video, audio and document resources related to the course can be obtained from the existing teaching course data set, which is existing data and will not be described in detail here.

[0054] Step 2: Obtain the triple representation and description of video knowledge, the triple representation and description of audio knowledge, and the triple representation and description of document knowledge from the video, audio, and document.

[0055] Specifically, a triple is represented by (,,), where S is the subject, P is the predicate, and O is the object; S and O are entities, and P is the relationship between entity S and entity O. For example, the triple (Xiaoming, born, Chengdu) describes: Xiaoming was born in Chengdu; S is Xiaoming, O is Chengdu, they are different entities, and P is the relationship between S and O.

[0056] For multimodal course resources, this embodiment extracts corresponding triple representations and descriptions from video, audio, and documents respectively. The following is a description of video, audio, and documents respectively:

[0057] (I) Regarding video resources

[0058] First, the video frames of the video resources are analyzed by CV (Computer Vision) technology to identify the object information in the teaching scene, such as students, teachers, desks, podiums, blackboards, projectors, computer desktops, etc. Then, according to the projector information and computer desktop information in the object information, the video is segmented to obtain projector segment group segments and computer desktop segment group segments.

[0059] Then, the text box of the above segment group segment and its corresponding spatial structure coordinates are identified by OCR (Optical Character Recognition) technology, for example: "{text:'I. Composition of Water','width':560,'top':36,'left':54,'height':74}" This means: the upper left corner of the screen is the coordinate origin, the left side of the text box is located at 54 pixels, the top is located at 36 pixels, the width is 560 pixels, and the height is 74 pixels. It contains the text 'I. Composition of Water'.

[0060] For the identified text box, the text box criticality scoring function is used:

[0061]

[0062] Score each text box. α, β, γ are weight coefficients, indicating the importance of each factor. i Represents the i-th text box; Indicates the distance of the i-th text box from the left side of the screen; Indicates the distance of the i-th text from the left side of the screen; x i Indicates the horizontal position of the upper left corner of the i-th text box; y i Indicates the vertical position of the upper left corner of the i-th text box; h i Indicates the height of the i-th text box; A indicates the width of the screen; B indicates the height of the screen.

[0063] Using the scores of all text boxes, the text box contents with scores higher than the preset threshold are stored as key text fragments in the key text box set T k :

[0064] T k ={T i |S i ≥t}

[0065] Among them, S i For the i-th text box T i The corresponding criticality score of , t is the criticality score threshold.

[0066] Finally, we use NLP (Natural Language Processing) technology to analyze the key text box set T k All key text segments in the video are obtained to obtain their semantic information, and after filtering out meaningless words such as modal particles and verbs, the filtered video information text group is obtained.

[0067] After obtaining the video information text group, the NER (Named Entity Recognition) technology is used to obtain various entity data from the video information text group, such as keyword information such as names of people, places, organizations, and object names. Then the relationship between each entity is analyzed through RE (Relation Extraction) technology, and finally the NLP technology is combined to construct a triple representation of video knowledge and its description (the description includes the triple's expression in the original text and its adjacent context). For example: (water, composition, hydrogen); (water, composition, oxygen)-water consists of oxygen and hydrogen. It should be added that when using NER technology to process the video information text group, the text will first be segmented, then the part of speech will be analyzed, and the final returned result will be marked with words, word frequency and part of speech, and then all nouns will be treated as entities according to the part of speech.

[0068] (II) Audio resources

[0069] In this embodiment, the audio is converted into text information through ASR (Automatic Speech Recognition) technology, and then the text information is corrected through NLP technology to correct text errors that occur in the ASR transcription, such as polyphones, homophones, modal particles, mispronunciations, etc., and finally a corrected audio information text group is obtained.

[0070] After obtaining the audio information text group, the NER technology is used to obtain various entity data from the audio information text group, such as keyword information such as names of people, places, organizations, and object names. The relationship between each entity is analyzed through RE technology, and finally the triple representation and description of audio knowledge are constructed in combination with NLP technology. It should be noted that when using NER technology to process the audio information text group, the text will first be segmented, and then the part of speech will be analyzed. The final result returned will be marked with words, word frequency, and part of speech, and then all nouns will be treated as entities according to the part of speech.

[0071] (III) Document resources

[0072] In actual applications, document resources include but are not limited to teaching plans, electronic teaching materials, document materials, etc., and document types include but are not limited to WORD, EXCEL, PPT, PDF, TXT, CHM, etc. Therefore, different knowledge extraction methods are proposed for different document resources in this embodiment.

[0073] For plain text document resources, such as TXT, plain text WORD, plain text PDF, plain text CHM, etc., you can directly obtain their text information through document tools.

[0074] For image version document resources, such as PDF files, we first use OCR to extract the text in the image, and use CV technology to perform layout analysis to understand the reading order of the text. Finally, we use NLP technology to analyze the semantics and merge them into a complete text information.

[0075] For mixed text and image document resources, such as mixed text and image WORD, mixed text and image PDF, as well as EXCEL, PPT, etc., first use the document converter to uniformly convert them into PDF in pure image format, and then use the processing method for image version document resources to complete the text information extraction;

[0076] Finally, the text information from the plain text document resources, the image version document resources, and the mixed text and image document resources are merged into a document information text group.

[0077] After obtaining the document information text group, various entity data are obtained from the document information text group using NER technology, such as keyword information such as names of people, places, organizations, and object names. The relationship between entities is then analyzed using RE technology, and finally the triple representation of document knowledge and its description is constructed in combination with NLP technology. It should be noted that when using NER technology to process the document information text group, the text will first be segmented, then the part of speech will be analyzed, and the results returned will be labeled with words, word frequency, and part of speech, and then all nouns will be treated as entities based on the part of speech.

[0078] Step 3: Perform entity disambiguation and reference disambiguation on the triple representation and description of video knowledge, the triple representation and description of audio knowledge, and the triple representation and description of document knowledge to obtain disambiguated knowledge triples and descriptions.

[0079] After completing the knowledge extraction of multimodal resources, disambiguation processing is required to complete entity alignment. In this embodiment, two processes are mainly performed: entity disambiguation and reference disambiguation. Specifically:

[0080] For entity disambiguation, this embodiment combines the three triple sets obtained in step 2, namely, the triple representation of video knowledge and its description, the triple representation of audio knowledge and its description, and the triple representation of document knowledge and its description, into the total triple set of the course Triad all , all corresponding entities are merged into the set EN all The specific entity disambiguation steps are as follows:

[0081] (1) Use the pre-trained embedding model to embed each triple and its description E i Vectorize it and get its vector value And record With E i Corresponding Entity EN i association.

[0082] (2) Based on the entity information obtained by NER technology in step 2, the entities are organized into different types of sub-entity sets, such as NI (organization noun), NL (location noun), NS (place noun), NT (time noun), NZ (other proper nouns), NN (general noun), ND (direction noun), and NH (person noun).

[0083] (3) In each type of sub-entity set, similarity calculation is performed separately:

[0084]

[0085] in, Represents the i-th entity EN i The associated triples and their descriptions E i The vector value of Represents the jth entity EN j The associated triples and their descriptions E j The vector value of .

[0086] (4) The similarity is higher than the preset threshold t s The two sets of vector values ​​are considered similar, that is, the two entities EN corresponding to these two sets of vector values i and EN j Similar, let ENi =EN j , merge two entities. In particular, when EN i and EN j If the text representation is different, then EN j The text representation is EN i For example, the average and mean are combined into the average, and the mean is recorded as the alias of the average. Then rewrite the entity EN j The corresponding original triple, EN i Replace EN in the original triple j .For example:

[0087] A. When EN j When it is in the S position of the triplet, it is rewritten as E′ j =(EN i ,P j ,EN oj ).

[0088] B. When EN j When it is in the O position of the triplet, it is rewritten as E' j =(EN sj ,P j ,EN i ).

[0089] Among them, E′ j represents the jth disambiguated triple, P j Represents the relation P,EN of the jth triple oj Indicates the entity at position O in the original triple, EN sj Represents the entity at position S in the original triple.

[0090] (5) The similarity is lower than the preset threshold t S The two sets of vectors are not similar.

[0091] After completing the calculation of the vector value similarity between all triples, the cleaning and filtering can be completed, and then the disambiguated triples are placed in the set E′ and the entities are placed in the set EN′.

[0092] For reference disambiguation, in this embodiment, an end-to-end model (a unified standard model based on a large amount of data calculation and training) is used to perform reference disambiguation on the triple set E′ and the entity set EN′ after entity disambiguation. The specific steps are as follows:

[0093] (1) Based on the entity information obtained by NER technology in step 2, the pronouns therein are saved as a pronoun set PR.

[0094] (2) Using the end2end model, we extract an entity, PR, from PR and EN′ respectively.T and EN′ I In the above example, the corresponding triple and description are input into the model, and the model will directly output a probability P, where P represents the pronoun entity PR. T Refers to the entity EN' I The internal logic of end2end can be roughly understood as follows: the unified model first converts the input into a vector representation through models such as BERT and Word2Vec, and then calculates the difference between the two sets of vectors based on the pre-trained parameters. Its symbolic mathematical formula is:

[0095] V fused =[V T ; V I ]∈R d+m

[0096] P(same entity|V fused )=σ(W·V fused +b)

[0097] Among them, V T PR for entity T The corresponding triples and the vectors describing the calculations by the unified model; V I The triple corresponding to entity EN′ is the vector calculated by the same model; R d+m is a vector with d+m dimensions in the real field; d is the first input V T The vector dimension of m is the second input V I The vector dimension of V fused is the fusion vector of the two input vectors; W is the weight parameter of the model; B is the bias parameter of the model; σ is the activation function; P represents PR T Refers to the probability of EN′.

[0098] (3) When the probability P is greater than the preset reference probability threshold t p When you can T Considered as a reference to entity EN'. Modify PR T The corresponding triples, the PR T Replaced with EN′.

[0099] (4) After completing the reference disambiguation, the disambiguated triples are placed into the triple set E″ and the entities are placed into the set EN″.

[0100] It should be noted that the order of the entity disambiguation and reference disambiguation processes proposed in this embodiment is not fixed. Entity disambiguation can be performed first, or reference disambiguation can be performed first. After the disambiguation process, the disambiguated knowledge triples and their descriptions are obtained.

[0101] In one embodiment, after obtaining the disambiguated knowledge triples and their descriptions, the existing teaching knowledge graph can be used to standardize the knowledge triples and their descriptions. The details are as follows:

[0102] The existing teaching knowledge graph is converted into a graph triple representation, and the graph triple and the knowledge triple obtained in the previous order are vectorized using an embedded model. Then, based on the similarity calculation, the knowledge node corresponding to the knowledge triple in the teaching knowledge graph is found. Then, large model tools such as LLM are used to compare the knowledge triple and its description with the description of the knowledge point in the teaching knowledge graph to determine whether there is an error. If there is a deviation, LLM is used to standardize the knowledge triple and its description through the context of the knowledge triple and the node information in the teaching knowledge graph. Finally, the standardized knowledge triple and its description are obtained.

[0103] Step 4: extract knowledge point fragments in the corresponding modal resources through the description of the triple set E″, and associate the knowledge point fragments in the multimodal resources with multiple triples corresponding to an entity obtained in entity disambiguation and coreference disambiguation through EN″.

[0104] In this embodiment, the description of the triple includes the triple's expression in the original text and its adjacent context. In practical applications, based on the sentence segmentation ability of NLP, several sentences before and after the original text of the knowledge triple can be extracted as the context of the knowledge point, preferably 2 to 3 sentences before and after.

[0105] Since the final triple set and its description are knowledge extracted from multimodal resources, the corresponding knowledge may exist in multiple modal resources at the same time. Therefore, it is possible to determine from the process of entity disambiguation and reference disambiguation that the disambiguated entity EN″ corresponds to multiple triples and their descriptions extracted from different modalities. Among them, for a certain entity, there is a video text context and an audio text context. At this time, it is necessary to determine whether the video text context and the audio text context come from the same audio video resource. If not, the video segment from the corresponding video start frame to the end frame when the video text context is identified is captured as the knowledge point segment of the original video, and the audio segment from the corresponding audio start time code to the end time code when the audio text context is identified is captured as the audio knowledge point segment; if so, only the video segment from the corresponding video start frame to the end frame is captured as the knowledge point segment of the original video.

[0106] For knowledge triples from different modal resources, it is necessary to associate different modal resources through entities to better display knowledge information. For example, video knowledge triples come from video text information, and video text information comes from OCR video frame recognition, and the corresponding video knowledge point fragments are obtained. Similarly, the audio knowledge triples also obtain corresponding audio knowledge point fragments, then these two video knowledge point fragments and audio knowledge point fragments can be marked and associated with the same knowledge point, becoming a multimodal resource for this knowledge point. Further document knowledge triples come from the text information of the document. The document from which it comes can also be associated with this knowledge point. The association process of this embodiment mainly relies on the results of entity disambiguation and reference disambiguation, such as EN i Come to video resources, EN j From the document resource, after disambiguation EN i =EN j This means that the video resource and document resource can be considered as related resources, and the connection point is the entity EN i In this way, video, audio and documents can complete the multi-modal connection through this knowledge point. Furthermore, based on the comparison with the existing knowledge graph specifications, this knowledge point can also be associated with the teaching knowledge picture, and the knowledge points can also be linked through the predecessor and successor relationships of each node in the teaching knowledge graph. In this way, the multi-modal resources related to each knowledge point can be connected in a three-dimensional way.

[0107] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program includes a program code for executing the method shown in the flowchart.

[0108] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0109] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0110] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0111] As another aspect, the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the general method for intelligently splitting knowledge points in the field of education described in the above embodiment.

[0112] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the general method for intelligently splitting knowledge points in the field of education described in the above embodiment.

[0113] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0114] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0115] For those skilled in the art, the specific meanings of the above terms in the present invention can be understood in specific situations; the drawings in the embodiments are used to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0116] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A general method for intelligently splitting knowledge points in the field of education, characterized in that: include: Get video, audio and document resources related to the course; Acquire triple representation and description of video knowledge, triple representation and description of audio knowledge, and triple representation and description of document knowledge from video, audio, and document; Perform entity disambiguation and reference disambiguation on the triple representation and description of video knowledge, the triple representation and description of audio knowledge, and the triple representation and description of document knowledge to obtain disambiguated knowledge triples and descriptions; The knowledge point fragments in the corresponding modal resources are intercepted through the description of the knowledge triples, and the knowledge point fragments in the multimodal resources are associated according to the entity disambiguation and reference disambiguation of the entities in the knowledge triples.

2. The general method for intelligently splitting knowledge points in the field of education according to claim 1 is characterized in that: It also includes, after obtaining the disambiguated knowledge triples and their descriptions, using the existing teaching knowledge graph to standardize the knowledge triples and their descriptions.

3. The general method for intelligently splitting knowledge points in the field of education according to claim 1 or 2, characterized in that: The triple representation of the video knowledge and the process of obtaining its description specifically include: Identify object information in the teaching scene from video resources, including projector information and computer desktop information; Segment the video according to the projector information and the computer desktop information to obtain projector segment group segments and computer desktop segment group segments; Identify text boxes and their corresponding spatial structure coordinates in the projector segment group segment and the computer desktop segment group segment; Scoring the text boxes and determining key text fragments with scores above a preset threshold; Use NLP technology to analyze key text fragments, obtain their semantic information and filter out meaningless text to obtain the filtered video information text group; Various entity data are obtained from the video information text group, and the relationship between each entity is analyzed to construct a triple representation of video knowledge and its description.

4. The general method for intelligently splitting knowledge points in the field of education according to claim 1 or 2, characterized in that: The triplet representation of the audio knowledge and the acquisition process of its description specifically include: Convert audio resources into text information, and use NLP technology to correct the text information to obtain a corrected audio information text group; Various entity data are obtained from the audio information text group, and the relationship between each entity is analyzed to construct a triple representation of audio knowledge and its description.

5. The general method for intelligently splitting knowledge points in the field of education according to claim 1 or 2, characterized in that: The triple representation of the document knowledge and the acquisition process of its description specifically include: For plain text document resources, extract text information through document tools; For image version document resources, we extract the text in the image through OCR, perform layout analysis to determine the reading information of the text, and then use NLP technology to analyze the semantics and merge them into a complete text information; For document resources that contain both text and images, they are first converted into image-version document resources, and then the text information is extracted using the processing method for image-version document resources. Merge text information from plain text document resources, image version document resources, and mixed image and text document resources into a document information text group; Various entity data are obtained from the document information text group, and the relationship between each entity is analyzed to construct a triple representation and description of the document knowledge.

6. The general method for intelligently splitting knowledge points in the field of education according to claim 1 is characterized in that: The entity disambiguation includes: Merge the triple representation and description of video knowledge, the triple representation and description of audio knowledge, and the triple representation and description of document knowledge into a total triple set, and obtain a corresponding entity set; Use the pre-trained embedded model to vectorize each triple and its description in the total triple set to obtain the corresponding vector value; Divide the entities in the entity combination into different types of sub-entity sets; In each type of sub-entity set, calculate the similarity between each triple and the vector value corresponding to its description; Two sets of vector values ​​with similarity higher than a preset threshold are considered similar and the two entities are merged; two sets of vector values ​​with similarity lower than a preset threshold are considered dissimilar; After completing the vector value similarity calculation between all triples, the set of triples and the corresponding entity set after entity disambiguation are obtained.

7. The general method for intelligently splitting knowledge points in the field of education according to claim 1 is characterized in that: The reference disambiguation includes: Store the pronoun entities in the entity type as a pronoun set; Use the end-to-end model to extract an entity from the pronoun set and entity set respectively, input its corresponding triple and description into the model, and the model will directly output a reference probability; When the reference probability is greater than a preset reference probability threshold, the pronoun is regarded as a reference to the corresponding entity, and the pronoun entity in the triplet where the pronoun is located is replaced with the corresponding entity; After completing the probability calculation between all triples and pronouns, the set of triples and the corresponding entity set after reference disambiguation are obtained.

8. The general method for intelligently splitting knowledge points in the field of education according to claim 1 is characterized in that: The method of extracting knowledge point fragments in the corresponding modal resource through the description of the knowledge triples specifically includes: Directly extract knowledge point fragments in the corresponding modal resource according to the description of the knowledge triples, wherein the description of the knowledge triples includes the expression of the triples in the original text and its adjacent context; wherein, if the description of the knowledge triples has both video text context and audio text context, determine whether they are from the same audio video resource, if not, extract the video fragment from the corresponding video start frame to the end frame when the video text context is identified as the knowledge point fragment of the original video, and extract the audio fragment from the corresponding audio start time code to the end time code when the audio text context is identified as the audio knowledge point fragment; if yes, only extract the corresponding video fragment from the video start frame to the end frame as the knowledge point fragment of the original video.

9. The general method for intelligently splitting knowledge points in the field of education according to claim 8, characterized in that: The associating the knowledge point fragments in the multimodal resource according to the entity disambiguation and coreference disambiguation situations of the entities in the knowledge triples specifically includes: In the process of entity disambiguation and reference disambiguation, if the entity of the knowledge triple has a corresponding disambiguated entity, the entity and the modal resource where the corresponding disambiguated entity is located are associated resources, and the corresponding knowledge point fragments are associated by using the entity as the association point.

10. The general method for intelligently splitting knowledge points in the field of education according to claim 2, characterized in that: The method of using the existing teaching knowledge graph to standardize the knowledge triples and their descriptions specifically includes: Convert the existing teaching knowledge graph into a graph triple representation; Use the embedded model to vectorize the graph triples and disambiguated knowledge triples and calculate the similarity; According to the similarity calculation results, find the knowledge points of the knowledge triples in the teaching knowledge graph; The big model tool is then used to compare the knowledge triples and their descriptions with the descriptions of the knowledge points in the teaching knowledge graph to determine whether there are any errors. If there are deviations, the knowledge triples and their descriptions are standardized to finally obtain the standardized knowledge triples and their descriptions.

Images