A teaching resource cloud platform

Abstract

The application relates to the field of cloud sharing, in particular to a teaching resource cloud platform. The application determines a text segment to be corrected in real time through a resource acquisition module, determines observation frame groups and observation areas by performing action capture analysis on video data according to a video analysis module, determines the effectiveness of the observation areas through a verification module and sets verification tags for each time domain, searches for corresponding homophonic audio time domains of the text segment to be corrected through a correction matching module, further determines a plurality of matching correction time domains corresponding to the text segment to be corrected, regenerates a plurality of temporary text segments based on the text segment to be corrected through a correction response module, verifies the overall semantics of each temporary text segment, and finally determines a corrected text segment. The application comprehensively determines actions and audio in video data, reduces the influence of accents, sets subtitles more in line with contexts and semantics, facilitates students to understand, improves the accuracy of subtitle setting in teaching video data, and improves learning experience.

Description

Technical Field

[0001] This invention relates to the field of cloud sharing, and more particularly to a cloud platform for teaching resources. Background Technology

[0002] With the deepening development of educational informatization, cloud platforms for teaching resources have made significant progress in the sharing and application of teaching resources. In existing technologies, such platforms typically adopt a centralized architecture, supporting users such as teachers or students to upload various digital teaching resources, including courseware, exercises, and micro-lecture videos. The platform uniformly stores, categorizes, and tags the uploaded resources, constructing a structured resource sharing library, thus becoming an important tool supporting the sharing of teaching resources and self-directed learning.

[0003] For example, Chinese Patent Publication No. CN110188546A discloses a teaching resource management method based on an information database platform, relating to the field of internet technology. The method includes: a teaching terminal receiving instructions, recording a teaching process to obtain teaching resources, and sending the teaching resources to a corresponding information database platform node; the information database platform node saving the teaching resources to the information database platform; a client receiving teaching resource query information, generating a teaching resource query request, and sending it to the corresponding information database platform node; the information database platform node searching for the corresponding teaching resources in the information database platform according to the query request and sending them to the client; and the client receiving the teaching resources sent by the information database platform node. In this application, based on the information database platform, not only is the copyright of each teaching resource protected, but subsequent queries are also facilitated, realizing the sharing of teaching resources among schools; simultaneously, it provides an effective data foundation for matters such as school management and the integration of educational resources.

[0004] However, the following problems still exist in the existing technology. The video teaching materials uploaded to the teaching resource cloud platform come from a wide range of sources. Some of the teachers in these videos have accents, which makes it difficult for students to understand them. Furthermore, there are issues with speech recognition and subtitle annotation when configuring subtitles, which can negatively impact students' learning of the video teaching materials. Summary of the Invention

[0005] To address this, the present invention provides a teaching resource cloud platform to overcome the problems in the prior art where video teaching materials uploaded to the teaching resource cloud platform come from a wide range of sources, and some video teaching materials contain teachers with accents, which can easily lead to misrecognition of speech and mislabeling of subtitles, thus affecting students' learning of the video teaching materials.

[0006] To achieve the above objectives, the present invention provides a teaching resource cloud platform, comprising: The resource acquisition module is used to acquire video data uploaded by the terminal, perform audio separation and textification, and lock the text segment to be corrected based on the semantic relevance of the obtained text segment. The video analysis module is used to perform motion capture analysis on the video data, including identifying detection targets in video frames, locking observation frame groups based on the movement mode of the detection targets, and determining the observation area of ​​each video frame in the observation frame group based on the position of the detection targets. The verification module is used to extract the text segment corresponding to the audio based on the time domain of the observation frame group, identify the keywords in the observation area corresponding to the video frame, verify the validity of the observation area based on the semantic correlation between the text segment and the keywords, and set a verification label for the time domain of the text segment according to the verification result. The correction matching module is used to determine the audio segment corresponding to the text segment to be corrected, retrieve several homophonic audio time domains within the complete time domain corresponding to the video data, and filter each homophonic audio time domain based on the verification label to determine several matching correction time domains corresponding to the text segment to be corrected. The correction response module is used to regenerate fuzzy keywords in the text segment to be corrected, determine several temporary text segments, verify the overall semantics of each temporary text segment based on the observation frame group corresponding to the correction time domain, and filter out the corrected text segment from the temporary text segments.

[0007] Furthermore, the process by which the resource acquisition module locates the text segment to be corrected based on the semantic relevance of the obtained text segments includes: Used to calculate the semantic relevance between each keyword in a text segment and the remaining keywords, in order to identify fuzzy keywords; Text segments containing ambiguous keywords are identified as text segments requiring correction. The fuzzy keywords are those whose semantic relevance is less than a predetermined semantic correction threshold.

[0008] Furthermore, the video analysis module for locking the observation frame group based on the movement pattern of the detected target includes, Identify the text outlines in the corresponding background area in consecutive video frames to determine the coordinate information of the detected target; Determine the time period that meets the coordinate interaction conditions, and then define the video frames within that time period as a video frame group. The detection target is a hand or a handheld object, and the coordinate interaction conditions include the detection target moving towards the text outline and pausing, or the detection target coinciding with the text outline.

[0009] Furthermore, the video analysis module is used to determine the observation area of ​​each video frame in the observation frame group based on the location of the detected target, including, Used to determine the key points of the detected target in each video frame, determine the position coordinates of the key points, and determine the relative positional relationship between the key points and the text outlines in the background area; If the key point is located within the text outline, then the area corresponding to the text outline in the same row is determined as the observation area; If the key point is located outside the text outline, then the region including at least one line of text outline is determined based on the position coordinates of the key point, and the region is determined as the observation area.

[0010] Furthermore, the verification module verifies the validity of the observation region based on the semantic relevance between the text fragment and the keywords, and sets verification labels for the time domain where the text fragment is located based on the verification results, including... Determine the average semantic correlation between the text fragment and each keyword within the observation area; If the average semantic relevance is greater than the predetermined semantic relevance threshold, the observation area is verified as valid, and a verification label is set for the text fragment.

[0011] Furthermore, the correction and matching module is used to retrieve several identical audio time domains within the complete time domain corresponding to the video data, including, The audio segments are divided according to the corresponding keywords to obtain several audio sub-segments corresponding to the keywords; The complete audio of the video data is retrieved to identify several segments that are identical to the audio sub-segments, and the time domain corresponding to each segment is determined as the same audio time domain.

[0012] Furthermore, based on the verification label matching of the text segment to be corrected, several matching correction time domains are included, Determine whether each of the homophonic audio time domains has a verification label set. If a verification label exists, then the corresponding homophonic audio time domain is determined as the matching correction time domain.

[0013] Furthermore, the correction response module is used to regenerate fuzzy keywords in the text segment to be corrected, including... Remove ambiguous keywords from the text segment and generate several temporary text segments based on the language model, retaining the remaining keywords and filling in the ambiguous keywords.

[0014] Furthermore, the correction response module verifies the overall semantics of each temporary text segment based on the observation frame group corresponding to the correction time domain, including: Determine each video frame in the corresponding observation frame group within each matching correction time domain; Identify the keywords within the observation range corresponding to each video frame and construct several matching word groups; Determine the semantic relevance between the temporary text segment and each matching phrase, sort the temporary text segments based on the semantic relevance, and select the temporary text segment corresponding to the maximum semantic relevance. Then, identify the temporary text segment as the correction text segment.

[0015] Furthermore, the resource acquisition module is also used to annotate time-series corresponding text segments or correct text segments in the video data.

[0016] Compared with existing technologies, this invention sets up a resource acquisition module, a video analysis module, a verification module, a correction matching module, and a correction response module. The resource acquisition module determines the text segment to be corrected in real time. The video analysis module performs motion capture analysis on the video data to determine the observation frame group and observation area. The verification module confirms the validity of the observation area and sets verification labels for each time domain. The correction matching module retrieves the corresponding homophonic audio time domains of the text segment to be corrected, and then filters them based on the verification labels to determine several matching correction time domains. The correction response module regenerates fuzzy keywords in the text segment to be corrected, determining several temporary text segments. The overall semantics of each temporary text segment is verified using the observation frame group corresponding to the matching correction time domains, ultimately determining the corrected text segment. This invention utilizes a comprehensive judgment of actions and audio in video data, reducing the impact of accents and setting subtitles that are more contextual and semantically appropriate, facilitating student understanding and improving the accuracy of subtitle settings in teaching video data, thus enhancing the learning experience.

[0017] In particular, this invention identifies the corresponding text segment to be corrected in video data and performs motion capture analysis on the video data to lock the observation frame group and observation area. In practice, video data is often video of teachers lecturing on a platform with a blackboard as the background. Due to lecturing habits, teachers usually add lecturing actions during the lecture, sometimes with linkage between the actions and the content on the blackboard. In this case, the semantic reference corresponding to the audio usually has a strong consistency with the semantic reference of the content recorded on the blackboard in the video frame. Based on this, this invention locks the observation frame group and observation area according to the movement mode of the detected target, thereby facilitating the identification of video frames whose semantic reference is relatively consistent with the semantic reference of the corresponding keywords in the video frame. This provides data support for the subsequent processing of the text segment to be corrected. Even if there is an accent problem and the recognized text is blurry, the keywords in the corresponding observation area of ​​the video frame can be extracted as a basis for processing the text segment to be corrected. Furthermore, by using the comprehensive judgment of actions and audio in the video data, the influence of accent can be reduced, and subtitles that are more in line with the context and semantics can be set to facilitate student understanding. At the same time, the accuracy of subtitle settings in teaching video data is improved, thus enhancing the learning experience.

[0018] In particular, the present invention verifies the observation area corresponding to the observation frame group and sets corresponding verification labels. The semantic reference of the audio in the time domain corresponding to the observation frame group is usually highly consistent with the semantic reference of the keywords in the video frame. The present invention further verifies the observation frame group to ensure that the observation frame group can reflect the above phenomenon and guarantee reliability.

[0019] In particular, this invention identifies several homophonic audio time domains and filters them based on verification tags. In practice, the text segments to be corrected are often caused by accent issues. Even if the audio is re-identified, differences in pronunciation or specific descriptions of local accents or dialectal features can lead to semantic mismatches in the overall text segments. In this case, this invention considers prioritizing the retrieval of homophonic audio time domains from complete video data and filtering out those with verified tags whose semantic orientation of the audio is highly consistent with the semantic orientation of the video frames. Subsequently, the corresponding keywords in the observation frame group in the matching correction time domain are used to process the text segments to be corrected, thereby understanding the true semantics of the text segments to be corrected and annotating them with subtitles. By comprehensively judging the actions and audio in the video data, the influence of accents is reduced, and subtitles that are more in line with the context and semantics are set, making it easier for students to understand. At the same time, the accuracy of subtitle settings in teaching video data is improved, enhancing the learning experience.

[0020] In particular, this invention verifies the overall semantics of each temporary text segment based on the observation frame group corresponding to the time domain of the correction. By first removing ambiguous keywords, several temporary text segments are generated. Although these temporary text segments are grammatically correct, they cannot represent the true semantics of the text segment to be corrected. Therefore, the keywords in the observation area of ​​the observation frame group are used to verify the overall semantics of the temporary text segments. The temporary text segments that are closest to the semantics of the text segment to be corrected are selected from the image frames as the corrected text segments to reflect the true semantics. Then, the actions and audio in the video data are used to make a comprehensive judgment, reducing the influence of accents, and setting subtitles that are more in line with the context and semantics, which is convenient for students to understand. At the same time, the accuracy of the subtitle settings in the teaching video data is improved, and the learning experience is enhanced. Attached Figure Description

[0021] Figure 1 A simplified diagram of the teaching resource cloud platform structure according to an embodiment of the invention; Figure 2 A logic block diagram for locking the text segment to be corrected in an embodiment of the invention; Figure 3 This is a logic block diagram of the locked observation frame group according to an embodiment of the invention; Figure 4 A logic block diagram for verifying the validity of the observation area and setting verification labels in an embodiment of the invention. Detailed Implementation

[0022] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0023] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0024] Please see Figure 1 As shown, it is a simplified structural diagram of a teaching resource cloud platform according to an embodiment of the invention. The teaching resource cloud platform of this embodiment of the invention includes: The resource acquisition module is used to acquire video data uploaded by the terminal, perform audio separation and textification, and lock the text segment to be corrected based on the semantic relevance of the obtained text segment. The video analysis module is used to perform motion capture analysis on the video data, including identifying detection targets in video frames, locking observation frame groups based on the movement mode of the detection targets, and determining the observation area of ​​each video frame in the observation frame group based on the position of the detection targets. The verification module is used to extract the text segment corresponding to the audio based on the time domain of the observation frame group, identify the keywords in the observation area corresponding to the video frame, verify the validity of the observation area based on the semantic correlation between the text segment and the keywords, and set a verification label for the time domain of the text segment according to the verification result. The correction matching module is used to determine the audio segment corresponding to the text segment to be corrected, retrieve several homophonic audio time domains within the complete time domain corresponding to the video data, and filter each homophonic audio time domain based on the verification label to determine several matching correction time domains corresponding to the text segment to be corrected. The correction response module is used to regenerate fuzzy keywords in the text segment to be corrected, determine several temporary text segments, verify the overall semantics of each temporary text segment based on the observation frame group corresponding to the correction time domain, and filter out the corrected text segment from the temporary text segments.

[0025] Specifically, the video data uploaded by the terminal is a real-scene teaching video with a blackboard background. Of course, the video data uploaded to the terminal may be of various types, such as teaching videos based on PPT or teaching videos based on electronic handouts. In practice, only the real-scene teaching videos with a blackboard background are processed. The uploaded video data can be classified and organized in advance before processing.

[0026] Specifically, there are no restrictions on the method of audio separation and textification. For example, after separating the audio track of the video, speech recognition can be performed to obtain several text segments corresponding to the time sequence of the video data. The text segments can be divided based on a single sentence, treating a single sentence as a text segment. This is existing technology and will not be elaborated further.

[0027] It should be noted that the multiple functional modules involved in this application are only a logical division based on the functions implemented according to the present invention, and are not a strict limitation on the physical structure; in practical applications, the above functional modules can be implemented by one or more integrated circuits, a processor executing program code in memory, or a combination of the above devices.

[0028] Specifically, please refer to Figure 2 The diagram shown is a logical block diagram of the method for locking the text segment to be corrected according to an embodiment of the invention. The process by which the resource acquisition module locks the text segment to be corrected based on the semantic relevance of the obtained text segment includes: Used to calculate the semantic relevance between each keyword in a text segment and the remaining keywords, in order to identify fuzzy keywords; Text segments containing ambiguous keywords are identified as text segments requiring correction. The fuzzy keywords are those whose semantic relevance is less than a predetermined semantic correction threshold.

[0029] Specifically, the keywords in the text segment can be obtained by word segmentation using a word segmentation tool, which will not be elaborated further here.

[0030] Specifically, the purpose of setting the semantic correction threshold is to reflect the semantic level under clear semantics. The semantic correction threshold is predetermined. Several typical corpora based on Mandarin are selected as samples, divided into text segments, and the semantic relevance between the keywords and the remaining keywords in each text segment is determined. The mean of the semantic relevance is calculated to reflect the semantic level corresponding to the typical corpora. For the semantic correction threshold, the standard can be appropriately relaxed, and the mean of the semantic relevance can be reduced to 0.7 to 0.9 times as the semantic correction threshold. In practice, 0.8 times is preferred.

[0031] Specifically, there is no limitation on the method of determining semantic relevance. Keywords can be vectorized and then the cosine similarity can be calculated. The cosine similarity can be used as the semantic relevance. Of course, other methods can also be used, as long as they reflect the degree of semantic relevance. This is existing technology and will not be elaborated further.

[0032] Specifically, please refer to Figure 3 As shown, Figure 3 This is a logic block diagram of a locked observation frame group according to an embodiment of the invention. The video analysis module is used to lock the observation frame group based on the movement pattern of the detected target, including: Identify the text outlines in the corresponding background area in consecutive video frames to determine the coordinate information of the detected target; Determine the time period that meets the coordinate interaction conditions, and then define the video frames within that time period as a video frame group. The detection target is a hand or a handheld object, and the coordinate interaction conditions include the detection target moving towards the text outline and pausing, or the detection target coinciding with the text outline.

[0033] Specifically, the purpose of setting the detection target as a hand or a handheld object is to observe the teacher's hand movements. Based on the coordinate interaction conditions, it is possible to filter out situations where the teacher's hand points to or follows the outline of the text, so as to reflect the explanation of the text outline. At this time, the semantics of the audio and the semantics of the text outline usually have a high correlation.

[0034] In practice, there are no restrictions on the method of identifying and detecting targets. For example, a target detection algorithm can be used to identify the corresponding preset target. Of course, other methods can also be used, which will not be elaborated here.

[0035] Specifically, it can determine whether the target is moving toward the text outline in real time based on the distance between the target and the text outline. If the distance continues to decrease, it is determined that the target is moving toward the text outline. Furthermore, it can determine whether the target has stopped based on its speed. If the speed of the target becomes 0, it is determined that a stop has occurred.

[0036] In practice, the detection target moving towards the text outline and pausing is considered to meet the coordinate interaction condition. The corresponding time period is from the moment the detection target starts moving to the moment the pause occurs.

[0037] This invention identifies the corresponding text segment to be corrected in video data and performs motion capture analysis on the video data to lock the observation frame group and observation area. In practice, video data often consists of lectures by teachers on a platform with a blackboard as the background. Due to their teaching habits, teachers usually add lecturing actions during the lecture, sometimes with linkage between the actions and the content on the blackboard. In this case, the semantic reference corresponding to the audio usually has a strong consistency with the semantic reference of the content recorded on the blackboard in the video frame. Based on this, this invention locks the observation frame group and observation area according to the movement mode of the detected target, thereby facilitating the identification of video frames whose semantic reference is relatively consistent with the semantic reference of the corresponding keywords in the video frame. This provides data support for subsequent processing of the text segment to be corrected. Even if there are accent problems and the recognized text is blurry, keywords in the corresponding observation area of ​​the video frame can be extracted as a basis for processing the text segment to be corrected. Furthermore, by using a comprehensive judgment of actions and audio in the video data, the influence of accents can be reduced, and subtitles that are more in line with context and semantics can be set to facilitate student understanding. At the same time, the accuracy of subtitle settings in teaching video data is improved, enhancing the learning experience.

[0038] Specifically, the video analysis module is used to determine the observation area of ​​each video frame in the observation frame group based on the location of the detected target, including, Used to determine the key points of the detected target in each video frame, determine the position coordinates of the key points, and determine the relative positional relationship between the key points and the text outlines in the background area; If the key point is located within the text outline, then the area corresponding to the text outline in the same row is determined as the observation area; If the key point is located outside the text outline, then the region including at least one line of text outline is determined based on the position coordinates of the key point, and the region is determined as the observation area.

[0039] During implementation, to facilitate judgment, key points are set for the detection target. For the case of the hand, the center of the hand can be set as the key point. For the handheld object, the outline point as far away from the hand as possible can be selected on the outline of the handheld object as the key point, so as to reflect the situation of the handheld teaching aid pointing to the outline of the text.

[0040] When the key point is located within the text outline, the directional relationship between the hand movement and the text outline is relatively clear. You can directly select the entire text outline of the line where the key point is located and construct a rectangular area whose width corresponds to the height of the text outline and whose length can include the entire text outline of the line as the observation area.

[0041] When the key point is outside the text outline, such as when the key point is between two lines of text, you can appropriately select the text outlines of the two lines above and below the key point to construct the observation area. If there is only text below or above the key point, then select the nearest line of text outline to construct the observation area. Constructing the observation region through the above process allows for the selection of text contours with high semantic relevance, reducing semantic interference from other text contours in subsequent analysis and improving reliability.

[0042] Specifically, please refer to Figure 4 The diagram shown is a logical block diagram of an embodiment of the invention for verifying the validity of an observation region and setting verification labels. The verification module verifies the validity of the observation region based on the semantic correlation between the text fragment and the keywords, and sets verification labels for the time domain where the text fragment is located based on the verification results. Determine the average semantic correlation between the text fragment and each keyword within the observation area; If the average semantic relevance is greater than the predetermined semantic relevance threshold, the observation area is verified as valid, and a verification label is set for the text fragment.

[0043] Specifically, the semantic association threshold is predetermined. Several typical video data based on Mandarin are pre-selected as samples, and several observation frame groups are extracted. The text segments corresponding to the audio in the time domain of the observation frame group are determined simultaneously to solve the mean semantic association degree between the keywords in the corresponding observation area of ​​the several text segments and the observation frame group. The mean of the mean semantic association degree is also solved. In implementation, a certain error can be allowed to avoid the semantic association threshold being too high and the verification pass rate being too low. The mean can be appropriately reduced, and the semantic association threshold is set between 0.6 times or 0.8 times the mean, preferably 0.7 times.

[0044] When the semantic relevance between a single text fragment and a keyword is determined, the semantic relevance between the keyword and each keyword in the text fragment can be calculated separately, and then the average value can be calculated as the semantic relevance. This will not be elaborated further.

[0045] This invention verifies the observation area corresponding to the observation frame group and sets corresponding verification labels. The semantic reference of the audio in the time domain corresponding to the observation frame group is usually highly consistent with the semantic reference of the keywords in the video frame. This invention further verifies the observation frame group to ensure that the observation frame group can reflect the above phenomenon and guarantee reliability.

[0046] Specifically, the correction and matching module is used to retrieve several identical audio time domains within the complete time domain corresponding to the video data, including, The audio segments are divided according to the corresponding keywords to obtain several audio sub-segments corresponding to the keywords; The complete audio of the video data is retrieved to identify several segments that are identical to the audio sub-segments, and the time domain corresponding to each segment is determined as the same audio time domain.

[0047] It is understandable that audio segments and text segments correspond to each other. Furthermore, after the text segments are segmented, several keywords are obtained, and each keyword corresponds to a different audio sub-segment. This will not be elaborated further.

[0048] Specifically, based on the verification label matching of the text segment to be corrected, several matching correction time domains are included, Determine whether each of the homophonic audio time domains has a verification label set. If a verification label exists, then the corresponding homophonic audio time domain is determined as the matching correction time domain.

[0049] This invention identifies several homophonic audio temporal domains and filters them based on verification tags. In practice, the text segments to be corrected are often caused by accent issues. Even after audio re-recognition, differences in pronunciation or specific descriptions of local accents or dialectal features can lead to semantic mismatches in the overall text segments. In this case, this invention prioritizes retrieving homophonic audio temporal domains from complete video data and filters out those with verified tags whose semantic orientation is highly consistent with the semantic orientation of the video frames. Subsequently, the corresponding keywords in the observation frame group within the matching correction temporal domain are used to process the text segments to be corrected, thereby understanding the true semantics of the text segments to be corrected and annotating them with subtitles. By comprehensively judging the actions and audio in the video data, the influence of accents is reduced, and subtitles that are more in line with the context and semantics are set, making it easier for students to understand. This also improves the accuracy of subtitle settings in teaching video data and enhances the learning experience.

[0050] Specifically, the correction response module is used to regenerate fuzzy keywords in the text segment to be corrected, including: Remove ambiguous keywords from the text segment and generate several temporary text segments based on the language model, retaining the remaining keywords and filling in the ambiguous keywords.

[0051] Specifically, there is no specific limitation on the type of language model. For example, an existing large natural language model can be used. The original corrected text fragment containing fuzzy keywords is input, and certain instructions are given, such as replacing the fuzzy keywords to generate a new sentence that conforms to the semantic environment with the original sentence length. Then, the new sentence is used as a temporary text segment.

[0052] In some possible implementations, the original fuzzy keywords can be replaced with homophones or synonyms of the fuzzy keywords to obtain several temporary text segments, which will not be elaborated here.

[0053] Specifically, the error correction response module verifies the overall semantics of each temporary text segment based on the observation frame group corresponding to the error correction time domain, including: Determine each video frame in the corresponding observation frame group within each matching correction time domain; Identify the keywords within the observation range corresponding to each video frame and construct several matching word groups; Determine the semantic relevance between the temporary text segment and each matching phrase, sort the temporary text segments based on the semantic relevance, and select the temporary text segment corresponding to the maximum semantic relevance. Then, identify the temporary text segment as the correction text segment.

[0054] This invention verifies the overall semantics of each temporary text segment based on the observation frame group corresponding to the time domain of the correction. By first removing ambiguous keywords, several temporary text segments are generated. Although these temporary text segments are grammatically correct, they may not represent the true semantics of the text segment to be corrected. Therefore, the keywords in the observation area of ​​the observation frame group are used to verify the overall semantics of the temporary text segments. The temporary text segments that are closest to the semantics of the text segment to be corrected are selected from the image frames as the corrected text segments to reflect the true semantics. Then, the actions and audio in the video data are used to make a comprehensive judgment, reducing the influence of accents, and setting subtitles that are more in line with the context and semantics, making it easier for students to understand. At the same time, the accuracy of the subtitle settings in the teaching video data is improved, thus improving the learning experience.

[0055] Specifically, the resource acquisition module is also used to annotate time-series corresponding text segments or correct text segments in the video data.

[0056] Understandably, for non-text segments, the text segments obtained after audio separation and textification are directly labeled in the corresponding video frames of the video data. For text segments to be corrected, if a corrected text segment can be output in the end, the corrected text segment is used to replace the text segment to be corrected in the corresponding video frames. If a corrected text segment cannot be output, the video frame corresponding to the text segment to be corrected can be recorded and manually labeled later.

[0057] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A teaching resource cloud platform, characterized in that, include: The resource acquisition module is used to acquire video data uploaded by the terminal, perform audio separation and textification, and lock the text segment to be corrected based on the semantic relevance of the obtained text segment. The video analytics module is used to perform motion capture analysis on the video data. This includes identifying a target in a video frame, locking an observation frame group based on the movement of the target, and determining the observation area of ​​each video frame in the observation frame group based on the position of the target. The verification module is used to extract the text segment corresponding to the audio based on the time domain of the observation frame group, identify the keywords in the observation area corresponding to the video frame, verify the validity of the observation area based on the semantic correlation between the text segment and the keywords, and set a verification label for the time domain of the text segment according to the verification result. The correction matching module is used to determine the audio segment corresponding to the text segment to be corrected, retrieve several homophonic audio time domains within the complete time domain corresponding to the video data, and filter each homophonic audio time domain based on the verification label to determine several matching correction time domains corresponding to the text segment to be corrected. The correction response module is used to regenerate fuzzy keywords in the text segment to be corrected, determine several temporary text segments, verify the overall semantics of each temporary text segment based on the observation frame group corresponding to the correction time domain, and filter out the corrected text segment from the temporary text segments.

2. The teaching resource cloud platform according to claim 1, characterized in that, The process by which the resource acquisition module locks the text segment to be corrected based on the semantic relevance of the obtained text segment includes: Used to calculate the semantic relevance between each keyword in a text segment and the remaining keywords, in order to identify fuzzy keywords; Text segments containing ambiguous keywords are identified as text segments requiring correction. The fuzzy keywords are those whose semantic relevance is less than a predetermined semantic correction threshold.

3. The teaching resource cloud platform according to claim 1, characterized in that, The video analysis module is used to lock the observation frame group based on the movement pattern of the detected target, including: Identify the text outlines in the corresponding background area in consecutive video frames to determine the coordinate information of the detected target; Determine the time period that meets the coordinate interaction conditions, and then define the video frames within that time period as a video frame group. The detection target is a hand or a handheld object, and the coordinate interaction conditions include the detection target moving towards the text outline and pausing, or the detection target coinciding with the text outline.

4. The teaching resource cloud platform according to claim 3, characterized in that, The video analysis module is used to determine the observation area of ​​each video frame in the observation frame group based on the location of the detected target. include, Used to determine the key points of the detected target in each video frame, determine the position coordinates of the key points, and determine the relative positional relationship between the key points and the text outlines in the background area; If the key point is located within the text outline, then the area corresponding to the text outline in the same row is determined as the observation area; If the key point is located outside the text outline, then the region including at least one line of text outline is determined based on the position coordinates of the key point, and the region is determined as the observation area.

5. The teaching resource cloud platform according to claim 1, characterized in that, The verification module verifies the validity of the observation region based on the semantic correlation between the text fragment and the keywords, and sets verification labels for the time domain where the text fragment is located based on the verification results. Determine the average semantic correlation between the text fragment and each keyword within the observation area; If the average semantic relevance is greater than the predetermined semantic relevance threshold, the observation area is verified as valid, and a verification label is set for the text fragment.

6. The teaching resource cloud platform according to claim 1, characterized in that, The correction and matching module is used to retrieve several identical audio temporal domains within the complete temporal domain corresponding to the video data, including, The audio segments are divided according to the corresponding keywords to obtain several audio sub-segments corresponding to the keywords; The complete audio of the video data is retrieved to identify several segments that are identical to the audio sub-segments, and the time domain corresponding to each segment is determined as the same audio time domain.

7. The teaching resource cloud platform according to claim 6, characterized in that, Based on the verification tag matching, several matching correction time domains corresponding to the text segment to be corrected are included, Determine whether each of the homophonic audio time domains has a verification label set. If a verification label exists, then the corresponding homophonic audio time domain is determined as the matching correction time domain.

8. The teaching resource cloud platform according to claim 1, characterized in that, The correction response module is used to regenerate fuzzy keywords in the text segment to be corrected, including... Remove ambiguous keywords from the text segment and generate several temporary text segments based on the language model, retaining the remaining keywords and filling in the ambiguous keywords.

9. The teaching resource cloud platform according to claim 8, characterized in that, The correction response module verifies the overall semantics of each temporary text segment based on the observation frame group corresponding to the correction time domain, including: Determine each video frame in the corresponding observation frame group within each matching correction time domain; Identify the keywords within the observation range corresponding to each video frame and construct several matching word groups; Determine the semantic relevance between the temporary text segment and each matching phrase, sort the temporary text segments based on the semantic relevance, and select the temporary text segment corresponding to the maximum semantic relevance. Then, identify the temporary text segment as the correction text segment.

10. The teaching resource cloud platform according to claim 1, characterized in that, The resource acquisition module is also used to annotate time-series corresponding text segments or correct text segments in the video data.

Images