AR-based intelligent lesson preparation method and system, and electronic equipment

Abstract

The invention relates to the technical field of intelligent teaching, in particular to an intelligent lesson preparation method and system based on AR and electronic equipment. The method comprises the steps of obtaining a current frame image of target data shot by an AR camera on a head-mounted AR device worn by a user; and performing OCR identification on the current frame image, extracting vocabularies of which the occurrence frequency is in the first preset rank in the current frame image, obtaining vocabularies of which the occurrence frequency is in the second preset rank in the target courseware found by the user, recording the vocabularies as a target vocabulary set, training the courseware intelligent index model, and obtaining the target vocabulary set in the subsequent lesson preparation process. And performing intelligent indexing of the courseware by using the courseware intelligent indexing model, and displaying the courseware output by the courseware intelligent indexing model in the head-mounted AR device of the user. When the user tries to extract or query specific courseware data, the query result is optimized according to the historical query behavior of the user or the relevance between the data, and the efficiency and the intelligent degree of the lesson preparation process are improved to a great extent.

Description

Technical Field

[0001] The present invention relates to the field of intelligent teaching technology, and in particular to an AR-based intelligent lesson preparation method and system, and electronic equipment. Background Art

[0002] With the rapid growth of digital information, data integration management has become an important challenge in the field of modern information technology. Traditional data storage and retrieval systems face many limitations when processing large-scale and diverse courseware data.

[0003] In the field of intelligent teaching, most existing systems lack effective tools to analyze and mine the potential correlation between target keywords and courseware data. They can usually only provide simple searches based on keywords, but cannot retrieve based on deeper connections and patterns between data, limiting the user's ability to quickly find target courseware from a large amount of courseware data. When users try to extract or query specific courseware data, existing systems often cannot optimize query results based on the user's historical query behavior or the correlation between data, which prevents users from quickly obtaining related and potentially more valuable courseware. This results in a large workload and low efficiency in lesson preparation.

[0004] Therefore, the prior art needs to be further developed. Summary of the invention

[0005] The purpose of the present invention is to overcome the above-mentioned technical deficiencies and provide an AR-based intelligent lesson preparation method and system, and electronic equipment to solve the problems existing in the prior art.

[0006] To achieve the above technical objectives, according to a first aspect of the present invention, the present invention provides an AR-based intelligent lesson preparation method, comprising:

[0007] S100, receiving courseware data from data sources of different types and formats, and constructing a courseware database; in the current lesson preparation process, obtaining the current frame image of the target data taken by the AR camera on the head-mounted AR device worn by the user; performing OCR recognition on the current frame image, extracting the words with the first preset ranking of the number of appearances in the current frame image, recording them as target keywords, and sorting the target keywords in the current frame image according to the number of appearances, and assigning preset weights to each target keyword in turn according to the preset weight list, and calculating the number of appearances of the target keywords of each courseware in the courseware database in turn, and according to the number of appearances of each target keyword and each The preset weight of the target keyword is used to score each courseware in the courseware database, and the courseware with the highest score is displayed on the user's head-mounted AR device. Human-computer interaction is performed with the user to confirm whether the target courseware is found. If so, the vocabulary ranked from the largest to the smallest in terms of the number of occurrences in the target courseware and ranked as the first or second preset rankings are obtained and recorded as the target vocabulary set. The target keywords and target vocabulary sets extracted in the current frame image during the lesson preparation process are recorded as the courseware index process data set, and a preset number of courseware index process data sets are collected, and the collected courseware index process data sets are divided into a training data set and a verification data set according to a first preset ratio;

[0008] S200, using the collected training data set to train the courseware intelligent indexing model, using the verification data set to optimize the courseware intelligent indexing model, optimizing the model parameters, and completing the courseware intelligent indexing model training;

[0009] S300. When the model training is completed, during the next lesson preparation process, obtain the current frame image of the target data taken by the AR camera on the head-mounted AR device worn by the user, perform OCR recognition on the current frame image, extract the target keywords in the current frame image, input the extracted target keywords into the courseware intelligent indexing model, and display the courseware output by the courseware intelligent indexing model on the user's head-mounted AR device.

[0010] Specifically, the target data includes at least one of the following:

[0011] The content of the textbooks needed for the current lesson preparation, the content of the teaching guidance documents needed for the current lesson preparation, and the content of the reference books needed for the current lesson preparation.

[0012] Specifically, the human-computer interaction with the user to confirm whether the target courseware is found includes:

[0013] A prompt signal is output to the user to indicate whether the target courseware has been found, and a feedback signal input by the user is obtained. The feedback signal input by the user includes a feedback signal confirming that the target courseware has been found and a feedback signal confirming that the target courseware has not been found. Whether the target courseware has been found is confirmed based on the feedback signal input by the user.

[0014] Specifically, the human-computer interaction with the user to confirm whether the target courseware is found also includes:

[0015] If the target courseware is not found, the current frame image of the target material taken by the AR camera on the head-mounted AR device worn by the user is re-acquired, and process S100 is re-executed.

[0016] Specifically, the method of using the collected training data set to train the courseware intelligent indexing model, using the verification data set to optimize the courseware intelligent indexing model, optimizing the model parameters, and completing the courseware intelligent indexing model training includes:

[0017] The training data set is input into the preset network layer in batches for training. The preset network layer includes a Transformer network layer. The Transformer network layer is used to predict the next courseware indexing process data set based on the current courseware indexing process data set by forward propagation, and then obtain the predicted loss value; the loss value of the preset network layer is calculated and input into the optimizer for optimization to determine the direction in which the parameter gradient of the courseware intelligent indexing model decreases fastest; the courseware intelligent indexing model performs back propagation according to the loss value and the parameter gradient of the model to optimize the parameters of the courseware intelligent indexing model.

[0018] Specifically, the method of using the collected training data set to train the courseware intelligent indexing model, using the verification data set to optimize the courseware intelligent indexing model, optimizing the model parameters, and completing the courseware intelligent indexing model training includes:

[0019] After each training, the verification data set is input into the preset network layer of the previous training in batches for model parameter verification, and cyclic training is performed, with the total number of training rounds set to the first preset round; the loss value of the preset network layer is recorded, and it is determined whether the loss value meets the first preset condition, and whether to end the training and output the model parameters based on the judgment result.

[0020] Specifically, the determining whether the loss value satisfies the first preset condition, and determining whether to end the training and output the model parameters according to the determination result, includes:

[0021] If the loss value meets the first preset condition, the training is terminated and the parameters of the current courseware intelligent indexing model are output; if the loss value does not meet the first preset condition, the training is continued;

[0022] The first preset condition includes:

[0023] After the second preset round of training, the loss values ​​obtained in the next round of training are greater than or equal to the loss values ​​that occurred during the second preset round of training.

[0024] Specifically, scoring each courseware in the courseware database according to the number of occurrences of each target keyword and the preset weight of each target keyword includes:

[0025] Select a courseware and select a target keyword from the courseware, calculate the product of the preset weight of the target keyword and the number of times it appears in the courseware, record it as the first score of the target keyword, calculate the sum of the first scores of all target keywords of the courseware, record it as the score of the courseware.

[0026] According to a second aspect of the present invention, there is provided an AR-based intelligent lesson preparation system, comprising:

[0027] The acquisition module is used to receive courseware data from data sources of different types and formats and build a courseware database; it is used to obtain the current frame image of the target data taken by the AR camera on the head-mounted AR device worn by the user during the current lesson preparation process;

[0028] The control module is used to perform OCR recognition on the current frame image, extract the vocabulary with the first preset ranking of the number of appearances in the current frame image, record it as the target keyword, and sort the target keywords in the current frame image according to the number of appearances, and assign the preset weight to each target keyword in turn according to the preset weight list, calculate the number of appearances of the target keywords of each courseware in the courseware database in turn, score each courseware in the courseware database according to the number of appearances of each target keyword and the preset weight of each target keyword, display the courseware with the highest score to the user's head-mounted AR device, perform human-computer interaction with the user, confirm whether the target courseware is found, and if so, obtain the vocabulary with the second preset ranking of the number of appearances in the target courseware from large to small, record it as the target vocabulary set, and extract the current frame image during the lesson preparation process. The target keywords and target vocabulary sets in the courseware indexing process data set are recorded as the courseware indexing process data set, a preset number of courseware indexing process data sets are collected, and the collected courseware indexing process data sets are divided into a training data set and a verification data set according to a first preset ratio; the courseware intelligent indexing model is trained using the collected training data set, and the courseware intelligent indexing model is optimized using the verification data set, the model parameters are optimized, and the courseware intelligent indexing model training is completed; when the model training is completed, in the next lesson preparation process, the current frame image of the target data taken by the AR camera on the head-mounted AR device worn by the user is obtained, OCR recognition is performed on the current frame image, the target keywords in the current frame image are extracted, the extracted target keywords are input into the courseware intelligent indexing model, and the courseware output by the courseware intelligent indexing model is displayed on the user's head-mounted AR device.

[0029] According to a third aspect of the present invention, there is provided an electronic device, comprising: a memory; and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above-mentioned AR-based intelligent lesson preparation method is implemented.

[0030] Beneficial effects:

[0031] In the course of preparing lessons, the present invention obtains the current frame image of the target data photographed by the AR camera on the head-mounted AR device worn by the user; performs OCR recognition on the current frame image, extracts the vocabulary with the first preset ranking of the number of occurrences in the current frame image, obtains the vocabulary with the second preset ranking of the number of occurrences in the target courseware found by the user according to the ranking from large to small, and records it as the target vocabulary set, and records the target keywords and target vocabulary set in the current frame image extracted in the course of preparing lessons as the courseware indexing process data set, and then trains the courseware intelligent indexing model, and then in the subsequent course preparation process, uses the courseware intelligent indexing model to perform intelligent indexing of the courseware, and displays the courseware output by the courseware intelligent indexing model to the user's head-mounted AR device. When the user attempts to extract or query specific courseware data, the query results are optimized according to the user's historical query behavior or the correlation between data, so as to quickly obtain related and potentially more valuable courseware, which greatly improves the efficiency, intelligence and accuracy of the course preparation process, greatly reduces the workload of course preparation, and greatly improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of an AR-based intelligent lesson preparation method provided in a specific embodiment of the present invention;

[0033] Figure 2 Schematic diagram of the system composition of the AR-based intelligent lesson preparation system provided in a specific embodiment of the present invention;

[0034] Figure 3 It is a structural diagram of the courseware intelligent indexing model provided in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is clearly and completely described below in conjunction with the accompanying drawings of the present invention. Based on the embodiments in this application, other similar embodiments obtained by ordinary technicians in this field without making creative work should all fall within the scope of protection of this application. In addition, the directional words mentioned in the following embodiments, such as "up", "down", "left", "right", etc., are only reference to the directions of the accompanying drawings. Therefore, the directional words used are used to illustrate rather than limit the invention.

[0036] The present invention will be further described below in conjunction with the accompanying drawings and preferred embodiments.

[0037] See also Figure 1 The present invention provides an AR-based intelligent lesson preparation method, comprising:

[0038] S100, receiving courseware data from data sources of different types and formats, and constructing a courseware database; in the current lesson preparation process, obtaining the current frame image of the target data taken by the AR camera on the head-mounted AR device worn by the user; performing OCR recognition on the current frame image, extracting the words with the first preset ranking of the number of appearances in the current frame image, recording them as target keywords, and sorting the target keywords in the current frame image according to the number of appearances, and assigning preset weights to each target keyword in turn according to the preset weight list, and calculating the number of appearances of the target keywords of each courseware in the courseware database in turn, and according to the number of appearances of each target keyword and each The preset weight of the target keyword is used to score each courseware in the courseware database, and the courseware with the highest score is displayed on the user's head-mounted AR device. Human-computer interaction is performed with the user to confirm whether the target courseware is found. If so, the vocabulary ranked from the largest to the smallest in terms of the number of occurrences in the target courseware and ranked as the first or second preset rankings are obtained and recorded as the target vocabulary set. The target keywords and target vocabulary sets extracted from the current frame image during the lesson preparation process are recorded as the courseware index process data set, a preset number of courseware index process data sets are collected, and the collected courseware index process data sets are divided into a training data set and a verification data set according to a first preset ratio.

[0039] Specifically, the target data includes at least one of the following:

[0040] The content of the textbooks needed for the current lesson preparation, the content of the teaching guidance documents needed for the current lesson preparation, and the content of the reference books needed for the current lesson preparation.

[0041] Specifically, the human-computer interaction with the user to confirm whether the target courseware is found includes:

[0042] A prompt signal is output to the user to indicate whether the target courseware has been found, and a feedback signal input by the user is obtained. The feedback signal input by the user includes a feedback signal confirming that the target courseware has been found and a feedback signal confirming that the target courseware has not been found. Whether the target courseware has been found is confirmed based on the feedback signal input by the user.

[0043] Specifically, the human-computer interaction with the user to confirm whether the target courseware is found also includes:

[0044] If the target courseware is not found, the current frame image of the target material taken by the AR camera on the head-mounted AR device worn by the user is re-acquired, and process S100 is re-executed.

[0045] It can be understood that the above process can realize the continuous updating of the data set of the courseware intelligent indexing model of the present invention and ensure the reliability of the data set. After the training of the courseware intelligent indexing model of the present invention is completed and put into use, the courseware intelligent indexing model of the present invention can still be trained with the continuously updated data set, so as to realize the continuous optimization of the courseware intelligent indexing model of the present invention and continuously improve the efficiency and accuracy of the courseware intelligent indexing of the present invention.

[0046] It should be noted that the steps before step S100 include:

[0047] A preset number, a first preset ratio, a first preset round, a second preset round, a preset weight list, a first preset ranking, and a second preset ranking are preset in the control module.

[0048] It can be understood that the preset number, the first preset ratio, the first preset round, the second preset round, the preset weight list, the first preset ranking, and the second preset ranking can be specifically set according to the actual needs of the user of the present invention. The present invention does not limit the specific values ​​of the above parameters, and there is no regularity in setting the specific values ​​of the above parameters, as long as it is applicable to the courseware management and courseware intelligent indexing method proposed in the present invention.

[0049] Preferably, the present invention sets the preset number to 500, the present invention sets the first preset ratio to 8:2, the present invention sets the first preset round to 10,000 times, the present invention sets the second preset round to 200, the present invention sets the preset weight list to {100, 90, 80, 70, 60, 50, 40, 30, 20, 10, 0}, the present invention sets the first preset ranking to the top 5, and the present invention sets the second preset ranking to the top 20. The above settings can further improve the performance of the model and the efficiency of model training, and greatly improve the accuracy and reliability of the prediction of the courseware intelligent indexing model of the present invention.

[0050] It can be understood that when the user is preparing lessons, if the user finds the target courseware through the vocabulary whose appearance number in the current frame image is before the first preset ranking, there is a certain coupling relationship between the vocabulary whose appearance number in the current frame image is before the first preset ranking and the vocabulary whose appearance number in the target courseware is ranked from large to small and ranked before the second preset ranking, that is, there is a certain coupling relationship between the current frame image of the target material taken by the AR camera on the user's head-mounted AR device and the content of the target courseware. Based on this, the present invention trains the courseware intelligent indexing model.

[0051] Specifically, receiving courseware data from data sources of different types and formats and constructing a courseware database includes:

[0052] Receive courseware data from data sources of different types and formats, support multiple data access protocols, and can handle a large number of concurrent data streams.

[0053] Specifically, the method supports multiple data access protocols, including HTTP / HTTPS, FTP, SFTP, WebDAV, and various database connection protocols, such as JDBC and ODBC.

[0054] Specifically, a concurrent processing mechanism is used to handle a large number of concurrent data streams. The engine is able to optimize data transmission, reduce latency, and ensure the stability and efficiency of data access even under high load conditions.

[0055] Specifically, the receiving of courseware data from data sources of different types and formats and constructing a courseware database includes:

[0056] Use OCR optical character recognition technology to scan paper courseware and obtain electronic courseware data.

[0057] Specifically, the data sources of different types and formats include:

[0058] File data source: including Excel data source and CSV data source, etc.

[0059] Database data sources: such as MySQL, MariaDB, Oracle, SQLServer, PostgreSQL, Greenplum, etc.

[0060] Big data engine data sources: such as Kylin, Cloudera Impala, Hive, MemSQL, Presto, MongoDB, etc.

[0061] Third-party interface data sources: such as OData, REST, and DataInsight.

[0062] S200, using the collected training data set to train the courseware intelligent indexing model, using the verification data set to optimize the courseware intelligent indexing model, optimizing the model parameters, and completing the courseware intelligent indexing model training.

[0063] Specifically, the method of using the collected training data set to train the courseware intelligent indexing model, using the verification data set to optimize the courseware intelligent indexing model, optimizing the model parameters, and completing the courseware intelligent indexing model training includes:

[0064] The training data set is input into the preset network layer in batches for training. The preset network layer includes a Transformer network layer. The Transformer network layer is used to predict the next courseware indexing process data set based on the current courseware indexing process data set by forward propagation, and then obtain the predicted loss value; the loss value of the preset network layer is calculated and input into the optimizer for optimization to determine the direction in which the parameter gradient of the courseware intelligent indexing model decreases fastest; the courseware intelligent indexing model performs back propagation according to the loss value and the parameter gradient of the model to optimize the parameters of the courseware intelligent indexing model.

[0065] It can be understood that Transformer is a deep learning model based on the self-attention mechanism, which consists of a multi-head attention mechanism and a feedforward neural network. It can process sequence data and capture long-distance dependencies. The Transformer model does not contain traditional convolutional layers, but processes each element in the input sequence through an attention layer. The typical structure of Transformer includes an encoder and a decoder, wherein the encoder is composed of multiple identical layers stacked together, and the decoder adds a self-attention layer on the basis of the encoder to process sequence generation tasks. The present invention uses Transformer to supplement global context information, enhances the model's capabilities in feature extraction and global understanding, greatly improves the performance of the model, and greatly reduces the computational cost.

[0066] Specifically, the preset network layer also includes a ResNet network layer, and the ResNet network layer is used to predict the output result according to the collected courseware index process data set, and the predicted result is a courseware image.

[0067] It should be noted here that ResNet is a deep convolutional neural network that solves the gradient vanishing problem in deep network training by introducing residual connections. The core component of ResNet is the residual block, which usually contains two or three convolutional layers and a jump connection, which allows the gradient to bypass these convolutional layers directly. The network structure of ResNet can be very deep. These networks extract the features of the target keywords and target vocabulary sets in the current frame image by stacking multiple residual blocks, and predict the output results through global average pooling and fully connected layers. The predicted results are courseware images.

[0068] See also Figure 3The present invention combines the ResNet network layer and the Transformer network layer. ResNet is good at extracting the local features of the target keywords and target vocabulary sets in the current frame image during the lesson preparation process, while Transformer can supplement the global context information. This combination can enhance the model's capabilities in feature extraction and global understanding, greatly improve the performance of the model, and greatly reduce the computational cost.

[0069] It can be understood that the ResNet network layer is used to predict the output result through the courseware index process data set collected in the training data set and the verification data set. The specific steps are:

[0070] (1) First, put the training data set into the Backbones network layer of the model. Backbone refers to a series of convolutional layers that constitute the skeleton of the neural network. The main function of these layers is to extract the features of the input data. The Backbone network is usually composed of multiple convolutional layers, pooling layers, and activation functions. It can extract meaningful feature representations from the original data. Backbones is composed of multiple Backbones, and the core network layer in Backbone is the Resnet network layer. After the data set is input into the model, the courseware index process data set will be input into Backbones for training;

[0071] (2) When constructing Backbone, the Resnet network layer was added. After the training data set was input, Resnet began to train layer by layer. The training extracted the features in the courseware indexing process data set and assisted Transformer in predicting the courseware.

[0072] Specifically, the method of using the collected training data set to train the courseware intelligent indexing model, using the verification data set to optimize the courseware intelligent indexing model, optimizing the model parameters, and completing the courseware intelligent indexing model training includes:

[0073] After each training, the verification data set is input into the preset network layer of the previous training in batches for model parameter verification, and cyclic training is performed, with the total number of training rounds set to the first preset round; the loss value of the preset network layer is recorded, and it is determined whether the loss value meets the first preset condition, and whether to end the training and output the model parameters based on the judgment result.

[0074] Specifically, after each training, the verification data set is input into the preset network layer of the previous training in batches to verify the model parameters, and the cyclic training is performed, including the following methods:

[0075] When the number of training times is not enough and the model needs to be optimized, the training can only be performed from the beginning. Therefore, the present invention develops a method for continuing the following training based on the previously trained model parameter file;

[0076] (1) Provide custom parameters, which can be set according to user needs to continue training based on any training parameters;

[0077] (2) Load the parameters of the custom model file;

[0078] (3) Put the verification data set into the custom parameter network layer for verification, obtain the loss value and record the model parameters of the model corresponding to the minimum loss value during the training process;

[0079] (4) Based on this set of model parameters, forward propagation training and back propagation are performed, and the optimizer optimizes the parameters to obtain a new set of data;

[0080] (5) Put the verification data into the new data network layer for inference to obtain the loss value;

[0081] (6) Perform cyclic training to obtain the best model and complete model optimization.

[0082] It is understandable that the training method adopted in the present invention is batch training, which means that each time the model parameters are updated, only a part of the samples in the validation data set is used, which is called a batch. The advantages of batch training are that it can reduce memory consumption, speed up training, increase randomness, and facilitate the generalization of the model.

[0083] It should be noted here that, in the present invention, batch_size=2, and the batch_size is the batch size. For example, when batch_size=2, it means that two courseware index process data sets are selected from the verification data set each time and put into the model for verification.

[0084] Specifically, the determining whether the loss value satisfies the first preset condition, and determining whether to end the training and output the model parameters according to the determination result, includes:

[0085] If the loss value meets the first preset condition, the training is terminated and the parameters of the current courseware intelligent indexing model are output; if the loss value does not meet the first preset condition, the training is continued;

[0086] The first preset condition includes:

[0087] After the second preset round of training, the loss values ​​obtained in the next round of training are greater than or equal to the loss values ​​that occurred during the second preset round of training.

[0088] Specifically, scoring each courseware in the courseware database according to the number of occurrences of each target keyword and the preset weight of each target keyword includes:

[0089] Select a courseware and select a target keyword from the courseware, calculate the product of the preset weight of the target keyword and the number of times it appears in the courseware, record it as the first score of the target keyword, calculate the sum of the first scores of all target keywords of the courseware, record it as the score of the courseware.

[0090] It can be understood that the present invention preferably sets the maximum round of cyclic training to 10,000 times, which can effectively ensure the accuracy of the model prediction of the present invention and record the loss value of the model training. The present invention sets the second preset round to 200, that is, when the loss value of the training no longer decreases within 200 rounds of training, the round with the smallest loss value in the 200 rounds of training is saved as the best verification round parameter, and the best model file is generated. The above setting makes the training rounds no longer rely on manual judgment, but uses the deep learning model for automatic training, which improves the accuracy of the model prediction, and automatically ends the training when the accuracy meets the requirements, which greatly saves the training time, and can also effectively prevent overfitting, which greatly improves the intelligence level of the present invention and the efficiency of model training.

[0091] Furthermore, the specific process of training the courseware intelligent indexing model of the present invention is as follows:

[0092] (1) Obtain the latent vector mean mu and logarithmic variance logvar in the forward propagation process of the courseware intelligent indexing model;

[0093] (2) Calculate a reasonable loss value representing the training effect:

[0094] Calculate the divergence: k1 = -0.5*(1+logvar-mu^2-(e^logvar));

[0095] Calculate the loss between the courseware index process data set and the predicted value in the validation data set:

[0096] loss = L1 + k1 * k1_weight;

[0097] Among them, loss is the loss value, k1 is the divergence, k1_weight is the proportion of k1, the value is 10, L1 is the absolute difference between the predicted value and the true value, and the calculation formula of L1 is as follows:

[0098]

[0099] Among them, A is the L1 loss in pytorch, which calculates the average or sum of the absolute differences between two tensors; A has a shape of N*C*H*W, which indicates that it is a four-dimensional tensor, N represents the number of samples (Batch Size), indicating that there are N data points, C represents the number of channels (Channels), H and W represent the height and width respectively, which are usually the two-dimensional spatial dimensions of the input features; P represents the padding value, which is a Boolean type, and the shape of P is c*h*w*1. The dimensions of P are the same as those of A in terms of channel, height, and width, but there is no sample number N, which means that P is a mask or padding information for a feature map of each sample, and the Boolean value of P may be used to mark whether the elements in A are filled (True) or valid (False).

[0100] (3) The loss value when using the validation data set to validate the current model network;

[0101] (4) If the validation loss of this round is less than that of the previous round, the best model parameters are replaced with the training model of this round;

[0102] (5) Put the rounds and loss values ​​into the dictionary one by one;

[0103] (6) Compare the loss value of the current round with the loss values ​​of the 200 rounds before this round (parameters are provided and can be set according to actual needs, the default is 200), that is, traverse the loss values ​​from epoch-200 to epoch-1 (epoch is the training round, and the comparison is performed when the epoch is greater than 200). If the loss values ​​of the current round of epoch are all greater than the loss values ​​from (epoch-200) to (epoch-1), then stop training.

[0104] (7) Save the best model parameters in the cyclic verification as a ckpt model file.

[0105] S300. When the model training is completed, during the next lesson preparation process, obtain the current frame image of the target data taken by the AR camera on the head-mounted AR device worn by the user, perform OCR recognition on the current frame image, extract the target keywords in the current frame image, input the extracted target keywords into the courseware intelligent indexing model, and display the courseware output by the courseware intelligent indexing model on the user's head-mounted AR device.

[0106] It can be understood that, in the course of preparing lessons, the present invention obtains the current frame image of the target data captured by the AR camera on the head-mounted AR device worn by the user; performs OCR recognition on the current frame image, extracts the vocabulary with the first preset ranking of the number of occurrences in the current frame image, obtains the vocabulary with the second preset ranking of the number of occurrences in the target courseware found by the user, and records it as the target vocabulary set; the target keywords and target vocabulary set extracted in the current frame image during the course preparation process are recorded as the courseware indexing process data set, and then trains the courseware intelligent indexing model, and then in the subsequent course preparation process, uses the courseware intelligent indexing model to perform intelligent indexing of the courseware, and displays the courseware output by the courseware intelligent indexing model to the user's head-mounted AR device. When the user attempts to extract or query specific courseware data, the query results are optimized according to the user's historical query behavior or the correlation between data, so as to quickly obtain related and potentially more valuable courseware, which greatly improves the efficiency, intelligence and accuracy of the course preparation process, greatly reduces the workload of course preparation, and greatly improves the user experience.

[0107] See also Figure 2 The present invention provides another embodiment, which provides an AR-based intelligent lesson preparation system, and the AR-based intelligent lesson preparation system includes:

[0108] The acquisition module 100 is used to receive courseware data from data sources of different types and formats to build a courseware database; to obtain the current frame image of the target data taken by the AR camera on the head-mounted AR device worn by the user during the current lesson preparation process;

[0109] The control module 200 is used to perform OCR recognition on the current frame image, extract the vocabulary with the first preset ranking of the number of appearances in the current frame image, record it as the target keyword, and sort the target keywords in the current frame image according to the number of appearances, and assign the preset weight to each target keyword in turn according to the preset weight list, calculate the number of appearances of the target keywords of each courseware in the courseware database in turn, score each courseware in the courseware database according to the number of appearances of each target keyword and the preset weight of each target keyword, display the courseware with the highest score to the user's head-mounted AR device, perform human-computer interaction with the user, confirm whether the target courseware is found, and if so, obtain the vocabulary with the second preset ranking of the number of appearances in the target courseware from large to small, record it as the target vocabulary set, and add the current frame image extracted during the lesson preparation process to the target vocabulary set. The target keywords and target vocabulary sets in the image are recorded as courseware indexing process data sets, a preset number of courseware indexing process data sets are collected, and the collected courseware indexing process data sets are divided into a training data set and a verification data set according to a first preset ratio; the collected training data set is used to train the courseware intelligent indexing model, and the verification data set is used to optimize the courseware intelligent indexing model, optimize the model parameters, and complete the courseware intelligent indexing model training; after the model training is completed, in the next lesson preparation process, the current frame image of the target data taken by the AR camera on the head-mounted AR device worn by the user is obtained, OCR recognition is performed on the current frame image, the target keywords in the current frame image are extracted, the extracted target keywords are input into the courseware intelligent indexing model, and the courseware output by the courseware intelligent indexing model is displayed on the user's head-mounted AR device.

[0110] It should be noted that, in the course of preparing lessons, the present invention obtains the current frame image of the target data captured by the AR camera on the head-mounted AR device worn by the user; performs OCR recognition on the current frame image, extracts the vocabulary with the first preset ranking of the number of occurrences in the current frame image, obtains the vocabulary with the second preset ranking of the number of occurrences in the target courseware found by the user, and records it as the target vocabulary set; the target keywords and target vocabulary set extracted in the current frame image during the course preparation process are recorded as the courseware indexing process data set, and then trains the courseware intelligent indexing model, and then in the subsequent course preparation process, uses the courseware intelligent indexing model to perform intelligent indexing of the courseware, and displays the courseware output by the courseware intelligent indexing model to the user's head-mounted AR device. When the user attempts to extract or query specific courseware data, the query results are optimized according to the user's historical query behavior or the correlation between data, so as to quickly obtain related and potentially more valuable courseware, which greatly improves the efficiency, intelligence and accuracy of the course preparation process, greatly reduces the workload of course preparation, and greatly improves the user experience.

[0111] In a preferred embodiment, the present application further provides an electronic device, the electronic device comprising:

[0112] A memory; and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the AR-based intelligent lesson preparation method is implemented. The computer device can be broadly a server, a terminal, or any other electronic device with necessary computing and / or processing capabilities. In one embodiment, the computer device may include a processor, a memory, a network interface, a communication interface, etc. connected via a system bus. The processor of the computer device can be used to provide necessary computing, processing and / or control capabilities. The memory of the computer device may include a non-volatile storage medium and an internal memory. An operating system, a computer program, etc. may be stored in or on the non-volatile storage medium. The internal memory may provide an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the computer device can be used to connect and communicate with external devices through a network. When the computer program is executed by the processor, the steps of the method of the present invention are performed.

[0113] The present invention may be implemented as a computer-readable storage medium having a computer program stored thereon, which causes the steps of the method of an embodiment of the present invention to be executed when executed by a processor. In one embodiment, the computer program is distributed on a plurality of computer devices or processors coupled to a network so that the computer program is stored, accessed and executed in a distributed manner by one or more computer devices or processors. A single method step / operation, or two or more method steps / operations, may be performed by a single computer device or processor or by two or more computer devices or processors. One or more method steps / operations may be performed by one or more computer devices or processors, and one or more other method steps / operations may be performed by one or more other computer devices or processors. One or more computer devices or processors may perform a single method step / operation, or perform two or more method steps / operations.

[0114] It will be appreciated by those skilled in the art that the method steps of the present invention may be performed by instructing related hardware such as a computer device or a processor through a computer program, and the computer program may be stored in a non-temporary computer-readable storage medium, which causes the steps of the present invention to be performed when the computer program is executed. Depending on the circumstances, any reference to memory, storage, database, or other media herein may include non-volatile and / or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0115] It can be understood that, in the course of preparing lessons, the present invention obtains the current frame image of the target data captured by the AR camera on the head-mounted AR device worn by the user; performs OCR recognition on the current frame image, extracts the vocabulary with the first preset ranking of the number of occurrences in the current frame image, obtains the vocabulary with the second preset ranking of the number of occurrences in the target courseware found by the user, and records it as the target vocabulary set; the target keywords and target vocabulary set extracted in the current frame image during the course preparation process are recorded as the courseware indexing process data set, and then trains the courseware intelligent indexing model, and then in the subsequent course preparation process, uses the courseware intelligent indexing model to perform intelligent indexing of the courseware, and displays the courseware output by the courseware intelligent indexing model to the user's head-mounted AR device. When the user attempts to extract or query specific courseware data, the query results are optimized according to the user's historical query behavior or the correlation between data, so as to quickly obtain related and potentially more valuable courseware, which greatly improves the efficiency, intelligence and accuracy of the course preparation process, greatly reduces the workload of course preparation, and greatly improves the user experience.

[0116] The various technical features described above can be combined arbitrarily. Although all possible combinations of these technical features are not described, any combination of these technical features should be considered to be covered by this specification as long as there is no contradiction in such combination.

[0117] The specific implementation of the present invention described above does not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. An intelligent lesson preparation method based on AR, characterized in that: The method comprises: S100, receiving courseware data from data sources of different types and formats, and constructing a courseware database; in the current lesson preparation process, obtaining the current frame image of the target data taken by the AR camera on the head-mounted AR device worn by the user; performing OCR recognition on the current frame image, extracting the words with the first preset ranking of the number of appearances in the current frame image, recording them as target keywords, and sorting the target keywords in the current frame image according to the number of appearances, and assigning preset weights to each target keyword in turn according to the preset weight list, and calculating the number of appearances of the target keywords of each courseware in the courseware database in turn, and according to the number of appearances of each target keyword and each The preset weight of the target keyword is used to score each courseware in the courseware database, and the courseware with the highest score is displayed on the user's head-mounted AR device. Human-computer interaction is performed with the user to confirm whether the target courseware is found. If so, the vocabulary ranked from the largest to the smallest in terms of the number of occurrences in the target courseware and ranked as the first or second preset rankings are obtained and recorded as the target vocabulary set. The target keywords and target vocabulary sets extracted in the current frame image during the lesson preparation process are recorded as the courseware index process data set, and a preset number of courseware index process data sets are collected, and the collected courseware index process data sets are divided into a training data set and a verification data set according to a first preset ratio; S200, using the collected training data set to train the courseware intelligent indexing model, using the verification data set to optimize the courseware intelligent indexing model, optimizing the model parameters, and completing the courseware intelligent indexing model training; S300. When the model training is completed, during the next lesson preparation process, obtain the current frame image of the target data taken by the AR camera on the head-mounted AR device worn by the user, perform OCR recognition on the current frame image, extract the target keywords in the current frame image, input the extracted target keywords into the courseware intelligent indexing model, and display the courseware output by the courseware intelligent indexing model on the user's head-mounted AR device.

2. The AR-based intelligent lesson preparation method according to claim 1, characterized in that: The target data includes at least one of the following: The content of the textbooks needed for the current lesson preparation, the content of the teaching guidance documents needed for the current lesson preparation, and the content of the reference books needed for the current lesson preparation.

3. The AR-based intelligent lesson preparation method according to claim 1, characterized in that: The human-computer interaction with the user to confirm whether the target courseware is found includes: A prompt signal is output to the user to indicate whether the target courseware has been found, and a feedback signal input by the user is obtained. The feedback signal input by the user includes a feedback signal confirming that the target courseware has been found and a feedback signal confirming that the target courseware has not been found. Whether the target courseware has been found is confirmed based on the feedback signal input by the user.

4. The AR-based intelligent lesson preparation method according to claim 1, characterized in that: The human-computer interaction with the user to confirm whether the target courseware is found also includes: If the target courseware is not found, the current frame image of the target material taken by the AR camera on the head-mounted AR device worn by the user is re-acquired, and process S100 is re-executed.

5. The AR-based intelligent lesson preparation method according to claim 1, characterized in that: The method of using the collected training data set to train the courseware intelligent indexing model, using the verification data set to optimize the courseware intelligent indexing model, optimizing the model parameters, and completing the courseware intelligent indexing model training includes: The training data set is input into the preset network layer in batches for training. The preset network layer includes a Transformer network layer. The Transformer network layer is used to predict the next courseware indexing process data set based on the current courseware indexing process data set by forward propagation, and then obtain the predicted loss value; the loss value of the preset network layer is calculated and input into the optimizer for optimization to determine the direction in which the parameter gradient of the courseware intelligent indexing model decreases fastest; the courseware intelligent indexing model performs back propagation according to the loss value and the parameter gradient of the model to optimize the parameters of the courseware intelligent indexing model.

6. The AR-based intelligent lesson preparation method according to claim 1, characterized in that: The method of using the collected training data set to train the courseware intelligent indexing model, using the verification data set to optimize the courseware intelligent indexing model, optimizing the model parameters, and completing the courseware intelligent indexing model training includes: After each training, the verification data set is input into the preset network layer of the previous training in batches for model parameter verification, and cyclic training is performed, with the total number of training rounds set to the first preset round; the loss value of the preset network layer is recorded, and it is determined whether the loss value meets the first preset condition, and whether to end the training and output the model parameters based on the judgment result.

7. The AR-based intelligent lesson preparation method according to claim 6, characterized in that: The determining whether the loss value satisfies the first preset condition, and determining whether to end the training and output the model parameters according to the determination result, includes: If the loss value meets the first preset condition, the training is terminated and the parameters of the current courseware intelligent indexing model are output; if the loss value does not meet the first preset condition, the training is continued; The first preset condition includes: After the second preset round of training, the loss values ​​obtained in the next round of training are greater than or equal to the loss values ​​that occurred during the second preset round of training.

8. The AR-based intelligent lesson preparation method according to claim 1, characterized in that: The step of scoring each courseware in the courseware database according to the number of occurrences of each target keyword and the preset weight of each target keyword includes: Select a courseware and select a target keyword from the courseware, calculate the product of the preset weight of the target keyword and the number of times it appears in the courseware, record it as the first score of the target keyword, calculate the sum of the first scores of all target keywords of the courseware, record it as the score of the courseware.

9. An AR-based intelligent lesson preparation system, characterized in that: include: The acquisition module is used to receive courseware data from data sources of different types and formats and build a courseware database; Used to obtain the current frame image of the target data captured by the AR camera on the head-mounted AR device worn by the user during the current lesson preparation process; Used in The control module is used to perform OCR recognition on the current frame image, extract the vocabulary with the first preset ranking of the number of appearances in the current frame image, record it as the target keyword, and sort the target keywords in the current frame image according to the number of appearances, and assign the preset weight to each target keyword in turn according to the preset weight list, calculate the number of appearances of the target keywords of each courseware in the courseware database in turn, score each courseware in the courseware database according to the number of appearances of each target keyword and the preset weight of each target keyword, display the courseware with the highest score to the user's head-mounted AR device, perform human-computer interaction with the user, confirm whether the target courseware is found, and if so, obtain the vocabulary with the second preset ranking of the number of appearances in the target courseware from large to small, record it as the target vocabulary set, and extract the current frame image during the lesson preparation process. The target keywords and target vocabulary sets in the courseware indexing process data set are recorded as the courseware indexing process data set, a preset number of courseware indexing process data sets are collected, and the collected courseware indexing process data sets are divided into a training data set and a verification data set according to a first preset ratio; the courseware intelligent indexing model is trained using the collected training data set, and the courseware intelligent indexing model is optimized using the verification data set, the model parameters are optimized, and the courseware intelligent indexing model training is completed; when the model training is completed, in the next lesson preparation process, the current frame image of the target data taken by the AR camera on the head-mounted AR device worn by the user is obtained, OCR recognition is performed on the current frame image, the target keywords in the current frame image are extracted, the extracted target keywords are input into the courseware intelligent indexing model, and the courseware output by the courseware intelligent indexing model is displayed on the user's head-mounted AR device.

10. An electronic device, characterized in that: include: Memory; and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the AR-based intelligent lesson preparation method according to any one of claims 1 to 8 is implemented.

Images