Learning object recognition methods, devices, equipment and storage media

Abstract

This disclosure relates to a method, apparatus, device, and storage medium for identifying learning objects. The method includes: acquiring a target page image containing learning objects; performing object recognition processing on the target page image using a pre-trained object recognition model to determine a first type of learning object; performing feature extraction processing on the target page image using a pre-trained feature vector extraction model to obtain a target feature vector and a second type of learning object; and determining the target type of the learning object in the target page image based on the first type, the target feature vector, and the second type. Therefore, for page images containing learning objects, the accuracy of learning object recognition is improved, especially for page images of poor quality, and / or page images containing multiple different learning objects with minor differences, which can accurately identify the type of learning object and then load an explanatory video matching the learning object, thereby enhancing students' learning interest.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for learning object recognition. Background Technology

[0002] To enhance students' learning interest when they are learning new knowledge in class or during their spare time, students can use a learning device to take pictures of book pages containing learning objects. The learning device can then identify the learning objects from the captured page images and load explanatory videos of those objects for explanation.

[0003] To identify learning objects from book page images, existing technologies employ multi-model fusion methods. However, in many cases, multi-model fusion methods fail to accurately identify the type of learning object, especially when the book page images captured by the learning device are of poor quality, and / or when the differences between different learning objects within the book page image are small. This leads to a mismatch between the loaded instructional video and the learning object. Therefore, it is necessary to provide a more accurate learning object identification method. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides a learning object recognition method, apparatus, device, and storage medium.

[0005] Firstly, this disclosure provides a method for learning object recognition, the method comprising:

[0006] Obtain the image of the target page containing the learning object;

[0007] Using a pre-trained object recognition model, object recognition processing is performed on the target book page image to determine the first type of the learned object;

[0008] Using a pre-trained feature vector extraction model, feature extraction processing is performed on the target page image to obtain the target feature vector and the second type of the learning object;

[0009] Based on the first type, the target feature vector, and the second type, the target type of the learning object in the target book page image is determined.

[0010] Secondly, this disclosure provides a learning object recognition device, the device comprising:

[0011] The image acquisition module is used to acquire images of target book pages containing learning objects;

[0012] The object recognition module is used to perform object recognition processing on the target book page image using a pre-trained object recognition model to determine the first type of the learned object;

[0013] The feature extraction module is used to perform feature extraction processing on the target page image using a pre-trained feature vector extraction model to obtain the target feature vector and the second type of the learning object;

[0014] The type determination module is used to determine the target type of the learning object in the target book page image based on the first type, the target feature vector, and the second type.

[0015] Thirdly, embodiments of this disclosure also provide an electronic device, the device comprising:

[0016] processor;

[0017] Memory, used to store executable instructions;

[0018] The processor is used to read executable instructions from memory and execute the executable instructions to implement the method provided in the first aspect above.

[0019] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon, wherein the storage medium stores the computer program, and when the computer program is executed by a processor, the processor causes the processor to implement the method provided in the first aspect above.

[0020] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0021] This disclosure discloses a learning object identification method, apparatus, device, and storage medium. The method involves acquiring a target page image containing learning objects; using a pre-trained object recognition model to perform object recognition processing on the target page image to determine a first type of learning object; using a pre-trained feature vector extraction model to perform feature extraction processing on the target page image to obtain a target feature vector and a second type of learning object; and based on the first type, the target feature vector, and the second type of learning object, determining the target type of the learning object in the target page image. Therefore, for page images containing learning objects, two different models can be used to perform object recognition and feature extraction on the target page image. By combining the object recognition results and feature extraction results, the final category of the learning object can be determined, improving the accuracy of learning object recognition. Even for page images of poor quality, and / or page images containing multiple different learning objects with minor differences, the type of learning object can be accurately identified, thereby loading an explanatory video matching the learning object and enhancing students' learning interest. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0023] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a learning object recognition method provided in an embodiment of this disclosure;

[0025] Figure 2 Provided for the embodiments of this disclosure Figure 1 A flowchart of the S140 process;

[0026] Figure 3 Provided for the embodiments of this disclosure Figure 2 A flowchart of the S220 process;

[0027] Figure 4 A flowchart illustrating another learning object recognition method provided in this embodiment of the disclosure;

[0028] Figure 5 This is a schematic diagram of the structure of a learning object recognition device provided in an embodiment of the present disclosure;

[0029] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0030] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0031] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0032] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0033] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0034] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0035] To improve the recognition accuracy of the learning objects, the following combines... Figures 1 to 4 The learning object identification method provided in the embodiments of this disclosure will be described. In the embodiments of this disclosure, the learning object identification method can be executed by an electronic device or a server. The electronic device may include devices with communication functions such as mobile phones, tablets, desktop computers, and laptops. The server may be a cloud server or a server cluster, or other devices with storage and computing functions. It should be noted that the following embodiments use an electronic device as the execution subject for illustrative explanation.

[0036] Figure 1 A flowchart illustrating a learning object recognition method provided in an embodiment of this disclosure is shown.

[0037] like Figure 1 As shown, the learning object recognition method may include the following steps.

[0038] S110. Obtain the target page image containing the learning object.

[0039] In this embodiment, when a student uses an electronic device (such as a learning machine) to take a picture of a page containing a learning object, the page image is used as the target page image, or the page image is preprocessed and the page is detected to generate the target page image, so as to identify the type of learning object from the target page image.

[0040] In some embodiments, the resolution of the target page image is greater than or equal to a preset resolution threshold, that is, the target page image is a high-resolution image. For such images, the difficulty of identifying the object category is relatively low, and type identification can be performed by methods such as multi-model fusion or the identification method described below in this embodiment.

[0041] In other embodiments, the resolution of the target page image is less than a preset resolution threshold, that is, the target page image is a low-resolution image. For such images, it is more difficult to identify the type of the learning object. It is difficult to accurately identify the type of the learning object from the target page image using the multi-model fusion method. Therefore, the identification method described below in this embodiment is required for type identification.

[0042] In some other embodiments, the target page image contains multiple learning objects, and the similarity between any two learning objects is less than or equal to a preset similarity threshold. That is, the multiple learning objects contained in the target page image are quite different. For such images, the difficulty of identifying the category of learning objects is relatively low, and type identification can be performed by methods such as multi-model fusion or the identification method described below in this embodiment.

[0043] In some other embodiments, the target page image contains multiple learning objects, and the similarity between any two learning objects is greater than a preset similarity threshold. That is, the multiple learning objects contained in the target page image are relatively similar. For such images, the identification of the learning object category is more difficult, and the identification method described below in this embodiment is required for type identification.

[0044] In other embodiments, the target page image may also be a low-resolution image, and may contain multiple learning objects with a similarity greater than a preset similarity threshold. The recognition difficulty of such images is further increased, and the recognition method described below in this embodiment is required for type recognition.

[0045] Optionally, the learning objects in the target page image can include, but are not limited to, celestial bodies, flowers, trees, or other objects, without any restrictions.

[0046] In this embodiment, optionally, S110 specifically includes: acquiring an initial page image; using a pre-trained page detection model to perform four-point bounding box detection processing on the initial page image to obtain the page position information of the initial page image; cropping candidate page images from the initial page image based on the page position information; correcting the initial shape of the pages in the candidate page images to the target shape to obtain the target page image.

[0047] Specifically, the initial page image is scaled to a fixed size and image data is standardized to obtain a preprocessed initial page image. Then, a page detection model is used to detect the position coordinates of the page content in the image from the preprocessed initial page image to obtain the page position information. Based on the page position information, images containing the page content are cropped from the preprocessed initial page image as candidate page images. Finally, the page shapes in the candidate page images are corrected to normal shapes to obtain the target page image.

[0048] The initial page image can be an image of a page captured using an electronic device such as a learning machine. The page detection model can include, but is not limited to, models such as Mask R-CNN networks. Page position information refers to the coordinate information of the page content within the image. Normally, a page is rectangular, but due to the way it was captured, it becomes trapezoidal, resulting in an initial trapezoidal shape in the image. However, the target shape of the page in the image is rectangular. To improve the recognition accuracy of the learning object, the trapezoidal page needs to be corrected to a rectangular shape.

[0049] Therefore, by performing page detection and proofreading processing on the captured images, the target page content containing normally shaped pages can be obtained, thereby improving the learning accuracy of the learning object.

[0050] S120. Using a pre-trained object recognition model, perform object recognition processing on the target page image to determine the first type of the learning object.

[0051] In some embodiments, for target page images containing multiple learning objects with small differences, and / or for target page images with low resolution, type recognition is more difficult. Therefore, the object recognition model needs to focus on strong features (e.g., difference features) of different objects and weaken weak features (e.g., identical or similar features) to improve category recognition accuracy. Accordingly, S120 specifically includes: performing self-attention processing on the target page image according to the target coefficients based on the self-attention network in the object recognition model to obtain a first self-attention feature; and performing classification processing on the first self-attention feature based on the classification network in the object recognition model to obtain a first type.

[0052] Specifically, focusing on the strong features of different objects can be understood as amplifying these strong features, using the aforementioned target coefficient as the weights of the strong features to amplify them. Conversely, weakening the weak features of different objects can be understood as processing these weak features into weaker features, again using the aforementioned target coefficient as the weights of the weak features to achieve the same effect. In this case, the target coefficient is Y = f(W / (max(SUM(W)-W,W))*X+b), where Y is the output value of the neurons in the self-attention network, f(*) is the activation function, W is the weights connecting the neurons in the self-attention network and the neurons in the classification network, X is the input value of the neurons in the self-attention network, and b is the bias value. Therefore, for target book page images where type recognition is challenging, the aforementioned target coefficient can amplify strong features and reduce weak features, thereby improving the accuracy of object recognition.

[0053] In other embodiments, for target page images containing multiple learning objects with significant differences, and / or for target page images with high resolution, the difficulty of type recognition is lower. The object recognition model can ensure category recognition accuracy without focusing on the strong features of different objects and weakening the weak features. Accordingly, S120 specifically includes: performing self-attention processing on the target page image according to the target coefficient based on the self-attention network in the object recognition model to obtain a first self-attention feature; and performing classification processing on the first self-attention feature based on the classification network in the object recognition model to obtain a first type.

[0054] In this case, the target coefficient is Y = f(W*X + b), where Y is the output value of the neuron in the self-attention network, f(*) is the activation function, W is the weight of the connection between the neuron in the self-attention network and the neuron in the classification network, X is the input value of the neuron in the self-attention network, and b is the bias value. Therefore, for target page images where type recognition is relatively easy, there is no need to amplify strong features and reduce weak features; the target coefficient alone is sufficient to ensure the accuracy of the target object's recognition.

[0055] Optionally, the object recognition model can include, but is not limited to, models such as ResNet50, VGG, and Vit. The self-attention network can specifically be the layer preceding the first classification network.

[0056] To improve the model's recognition performance, before inputting the target page image into the object recognition model, preprocessing operations such as scaling to a standard size and image data standardization can be performed on the target page image. This allows the object recognition model to perform self-attention processing on the preprocessed target page image, and then use the first classification network in the object recognition model to determine the classification label of the learned object, and take the meaning represented by the classification label as the first type.

[0057] In summary, using object recognition models with different weights for type recognition of target book page images with varying recognition difficulty is suitable for class recognition of images in different scenarios.

[0058] S130. Using a pre-trained feature vector extraction model, feature extraction processing is performed on the target page image to obtain the target feature vector and the second type of the learning object.

[0059] In some embodiments, for target page images containing multiple learning objects with small differences, and / or for target page images with low resolution, type recognition is more difficult. Therefore, the object recognition model needs to focus on the strong features of different objects and weaken the weak features to improve category recognition accuracy. Accordingly, S130 specifically includes: performing self-attention processing on the target page image according to the target coefficients based on the self-attention network in the feature vector extraction model to obtain second self-attention features; and performing classification processing on the second self-attention features based on the classification network in the feature vector extraction model to obtain the target feature vector and the second type of the learning object.

[0060] In this case, the target coefficient is Y = f(W / (max(SUM(W)-W,W))*X+b), where Y is the output value of the neuron in the self-attention network, f(*) is the activation function, W is the weight of the connection between the neuron in the self-attention network and the neuron in the classification network, X is the input value of the neuron in the self-attention network, and b is the bias value. Specifically, the above target coefficient is used as the weight of strong features to amplify them, and the above target coefficient is used as the weight of weak features to process them into weaker features. Therefore, for target book page images that are difficult to identify by type, the above target coefficient can amplify strong features and reduce weak features, thereby improving the recognition accuracy of the learned object.

[0061] In other embodiments, for target page images containing multiple learning objects with significant differences, and / or for target page images with high resolution, type recognition is less difficult. The object recognition model does not need to focus on the strong features of different objects and weaken the weak features to ensure the accuracy of category recognition. Accordingly, S130 specifically includes: performing self-attention processing on the target page image according to the target coefficient based on the self-attention network in the feature vector extraction model to obtain a second self-attention feature; and performing classification processing on the second self-attention feature based on the classification network in the feature vector extraction model to obtain the target feature vector and the second type of the learning object.

[0062] In this case, the second coefficient is Y = f(W*X + b), where Y is the output value of the neuron in the self-attention network, f(*) is the activation function, W is the weight of the connection between the neuron in the self-attention network and the neuron in the classification network, X is the input value of the neuron in the self-attention network, and b is the bias value. Therefore, for target page images where type recognition is relatively easy, there is no need to amplify strong features and reduce weak features; the recognition accuracy of the learned object can be guaranteed based on the above target coefficient.

[0063] Optionally, the feature vector extraction model can include, but is not limited to, models such as EfficientNet, and the feature vector extraction model differs from the object recognition model. Specifically, the self-attention network can be the preceding layer of the second classification network.

[0064] To improve the model's recognition performance, before inputting the target page image into the feature vector extraction model, preprocessing operations such as scaling to a standard size and image data standardization can be performed on the target page image. This allows the feature vector extraction model to perform self-attention processing on the preprocessed target page image, and then use the second classification network in the feature vector extraction model to determine the classification label and target feature vector of the learning object, and use the meaning represented by the classification label as the second type.

[0065] In summary, using object recognition models with different weights for type recognition of target book page images with varying recognition difficulty is suitable for class recognition of images in different scenarios.

[0066] S140. Based on the first type, the target feature vector, and the second type, determine the target type of the learning object in the target book page image.

[0067] In some embodiments, the target type is determined by comparing whether the first type and the second type are the same.

[0068] In other embodiments, the target type is determined by comparing the label type carried by the target feature vector with a first type.

[0069] In some other embodiments, the second type is determined by comparing the label type carried by the target feature vector with the second type.

[0070] In some other embodiments, the target type is determined by comparing the label type, the first type, and the second type carried by the target feature vector.

[0071] This disclosure provides a method for identifying learning objects. The method involves acquiring a target page image containing learning objects; using a pre-trained object recognition model to perform object recognition processing on the target page image to determine a first type of learning object; using a pre-trained feature vector extraction model to perform feature extraction processing on the target page image to obtain a target feature vector and a second type of learning object; and based on the first type, the target feature vector, and the second type, determining the target type of the learning object in the target page image. Therefore, for page images containing learning objects, two different models can be used to perform object recognition and feature extraction on the target page image. By combining the object recognition results and feature extraction results, the final category of the learning object can be determined, improving the accuracy of learning object recognition. Even for page images of poor quality, and / or page images containing multiple different learning objects with minor differences, the type of learning object can still be accurately identified, thereby loading an explanatory video matching the learning object and ensuring the student's learning effectiveness.

[0072] Furthermore, after determining the target type, the method also includes:

[0073] S150. Based on the target type, load and play the instructional video of the learning object from a video library containing multiple learning objects.

[0074] Specifically, the target type is matched with the video type of each explanatory video in the video library to load the explanatory video of the learning object from the video library containing multiple learning objects, and play the explanatory video so that students can master and understand the learning object by watching the explanatory video, thereby enhancing students' learning interest.

[0075] In another embodiment of this disclosure, in order to improve the efficiency of target type determination, multiple variables to be traversed that are highly similar to the target feature vector are first determined, and then the target type is determined by combining the label type, first type and second type carried by the multiple variables to be traversed respectively.

[0076] For ease of understanding, Figure 2 The specific implementation process of S140 above is shown in the figure. Figure 2 The method for determining the target type includes the following steps.

[0077] S210. Search for multiple vectors to be traversed from the pre-built feature vector search library that have a similarity to the target feature vector that satisfies the first similarity condition.

[0078] In this embodiment, the target feature vector is sent to a feature vector search library for querying, so as to find multiple feature vectors that are most similar to the target feature vector and use them as vectors to be traversed.

[0079] The first similarity condition indicates that the similarity between the target feature vector and the feature vector in the feature vector search library is greater than a preset threshold, or the first similarity condition indicates that the similarity between the target feature vector and the feature vector in the feature vector search library is the similarity at a preset sorting position.

[0080] Specifically, the similarity between the target feature vector and each feature vector in the feature vector search library is calculated, and the similarity is sorted. The top N similarity vectors are selected, or the M similarity vectors with similarity greater than a preset threshold are selected. The feature vectors corresponding to the N or M similarity vectors are then used as multiple vectors to be traversed. Optionally, N and M can be 5 or other values.

[0081] Optionally, the feature vector search library may include, but is not limited to, the ElasticSearch database (ES database) and the cloud-native vector database (Milvus).

[0082] S220. Determine the target type based on the label type, first type, and second type carried by the multiple vectors to be traversed.

[0083] For ease of understanding, Figure 3 The specific implementation process of S220 above is shown in the figure. Figure 3 The method for determining the target type includes the following steps.

[0084] S310. Determine whether the first type and the second type are consistent.

[0085] In this embodiment, the target type is first determined based on the first type and the second type output by the two models. Specifically, it is determined whether the first type and the second type are consistent. If they are consistent, S320 is executed; otherwise, S330 is executed.

[0086] S320, Use the first type as the target type.

[0087] In other cases, if the first type is the same as the second type, the second type can be directly used as the target type.

[0088] S330. Select the vector with the highest similarity from multiple vectors to be traversed, and use it as the target similarity variable of the target feature vector.

[0089] Specifically, the similarity between each vector to be traversed and the target feature vector is calculated. Then, the similarity of the multiple vectors to be traversed is sorted, and the feature vector with the highest similarity is selected as the target similarity vector.

[0090] S340. Determine whether the label types carried by the first type and the target similarity vector are consistent.

[0091] In this embodiment, the target similarity vector carries its corresponding label type, which can also be used to determine the target type. Specifically, when the first type and the second type are inconsistent, it is further determined whether the first type and the label type carried by the target similarity vector are consistent. If they are consistent, S350 is executed; otherwise, S360 is executed.

[0092] S350, Use the first type as the target type.

[0093] S360. Search for multiple first candidate feature vectors from multiple vectors to be traversed that are consistent with the label type carried by the target feature vector, and search for multiple second candidate feature vectors from multiple vectors to be traversed that are consistent with the first type.

[0094] Understandably, when the label types carried by the first type and the target similarity vector are inconsistent, the reliability of determining the target type based on the target similarity vector is poor. Therefore, it is necessary to select other reliable feature vectors from multiple vectors to be traversed. Specifically, the multiple vectors to be traversed are traversed again, and it is determined whether the label type carried by the current traversed vector is consistent with the label type carried by the target feature vector. If they are consistent, the current traversed vector is used as the first candidate feature vector. It is also determined whether the label type carried by the current traversed vector is consistent with the first type. If they are consistent, the current traversed vector is used as the second candidate feature vector, thus obtaining multiple first candidate feature vectors and multiple second candidate feature vectors.

[0095] S370. Increase the similarity value of each first candidate feature vector by a first value, and increase the similarity value of each second candidate feature vector by a second value.

[0096] To improve the reliability of determining the target type based on multiple first candidate feature vectors and multiple second candidate feature vectors, a predetermined empirical value is obtained, which includes a first value S1 and a second value S2. The similarity corresponding to each first candidate feature vector is increased by S1, and the similarity corresponding to each second candidate feature vector is increased by S2, thereby increasing the similarity of each candidate feature.

[0097] S380. Select the traversal vector with the highest similarity from the similarity between the first candidate feature vector after increasing the first value and the second candidate feature vector after increasing the second value, and take the label type carried by the traversal vector with the highest similarity as the target type.

[0098] Specifically, the first candidate feature vector after increasing the first value and the second candidate feature vector after increasing the second value are sorted in descending order or ascending order. Based on the sorting results, the variable with the highest similarity is selected as the target type, and the label type carried by the vector with the highest similarity is used as the target type.

[0099] Therefore, by combining the label type, first type, and second type carried by the target similarity vector, the target type can be comprehensively determined, which improves the accuracy of the target type determination. When the reliability of the target type determination method based on the target similarity vector is poor, the inaccuracy of the original target type determination can be corrected by increasing the first candidate feature vector and the second candidate feature vector.

[0100] In another embodiment of this disclosure, the first type can be directly used as the target type, or the target type can be determined based solely on the target feature vector and the second type, or the target type can be determined by combining the first type, the second type, and the target feature vector.

[0101] Figure 4 A logical schematic diagram of another learning object recognition method provided in an embodiment of this disclosure is shown.

[0102] like Figure 4 As shown, the learning object recognition method may include the following steps.

[0103] S410. Obtain the target page image containing the learning object.

[0104] S420. Using a pre-trained object recognition model, perform object recognition processing on the target page image to determine the first type of the learning object.

[0105] S410 to S420 are similar to S110 to S120, and will not be described in detail here.

[0106] S430, Use the first type as the target type.

[0107] In this embodiment, in order to improve the efficiency of type recognition, the first type can be directly used as the target type after obtaining the first type.

[0108] S440. Using a pre-trained feature vector extraction model, feature extraction processing is performed on the target page image to obtain the target feature vector and the second type of the learning object.

[0109] S440 is similar to S130, so it will not be described in detail here.

[0110] S450. Search for multiple vectors to be traversed from the pre-built feature vector search library that have a similarity to the target feature vector that satisfies the first similarity condition.

[0111] S450 is similar to S210, so it will not be described in detail here.

[0112] S460. Select the vector with the highest similarity from multiple vectors to be traversed, and use it as the target similarity variable of the target feature vector.

[0113] S460 is similar to S330, so it will not be described in detail here.

[0114] S470. Determine whether the label type carried by the target similarity variable is consistent with the second type.

[0115] In this embodiment, the target similarity vector carries its corresponding label type, which can also be used to determine the target type. Specifically, it determines whether the label type carried by the target similarity variable is consistent with the second type. If they are consistent, then S480 is executed; otherwise, S490 is executed.

[0116] S480. Use the label type carried by the target similarity variable as the target type.

[0117] S490. Search for multiple first candidate feature vectors from multiple vectors to be traversed that are consistent with the label type carried by the target feature vector.

[0118] Understandably, when the label type carried by the target similarity variable is inconsistent with the second type, the reliability of determining the target type based on the target similarity vector is poor. Therefore, it is necessary to select other reliable feature vectors from multiple vectors to be traversed. Specifically, the multiple vectors to be traversed are traversed again, and it is determined whether the label type carried by the current traversed vector is consistent with the label type carried by the target feature vector. If they are consistent, the current traversed vector is taken as the first candidate feature vector.

[0119] S491. Increase the similarity value of each first candidate feature vector by the first value.

[0120] To improve the reliability of determining the target type based on multiple first candidate feature vectors, a predetermined empirical value is obtained as the first value S1. The similarity corresponding to each first candidate feature vector is increased by S1, thereby increasing the similarity of each candidate feature.

[0121] S492. Select the traversable vector with the highest similarity from the similarity of the first candidate feature vector after increasing the first value, and take the label type carried by the traversable vector with the highest similarity as the target type.

[0122] Specifically, the first candidate feature vectors after the first value is increased are sorted in descending order or ascending order. Based on the sorting results, the variable with the highest similarity is selected as the target type, and the label type carried by the vector with the highest similarity is used as the target type.

[0123] Therefore, when the reliability of determining the target type based on the target similarity vector is poor, the original inaccurate target type determination can be corrected by increasing the first candidate feature vector.

[0124] S493. Based on the first type, the target feature vector, and the second type, determine the target type of the learning object in the target book page image.

[0125] S493 is similar to S140; see details below. Figure 2 and Figure 3 The description in the text.

[0126] In summary, the target type can be determined directly by the first type, or by combining the target feature vector and the second type, or by combining the first type, the second type, and the target feature vector.

[0127] This disclosure also provides a learning object recognition device for implementing the above-described learning object recognition method, which is described below in conjunction with... Figure 5 The following explanation is provided. In this embodiment, the learning object identification device can be an electronic device or a server. The electronic device can include devices with communication functions such as mobile phones, tablets, desktop computers, and laptops. The server can be a cloud server or server cluster, or other devices with storage and computing functions.

[0128] Figure 5 A schematic diagram of the structure of a learning object recognition device provided in an embodiment of this disclosure is shown.

[0129] like Figure 5 As shown, the learning object recognition device 500 may include:

[0130] Image acquisition module 510 is used to acquire images of target book pages containing learning objects;

[0131] The object recognition module 520 is used to perform object recognition processing on the target book page image using a pre-trained object recognition model to determine the first type of the learning object;

[0132] Feature extraction module 530 is used to perform feature extraction processing on the target page image using a pre-trained feature vector extraction model to obtain target feature vectors and a second type of learning object;

[0133] The type determination module 540 is used to determine the target type of the learning object in the target book page image based on the first type, the target feature vector, and the second type.

[0134] This disclosure provides a learning object recognition device that acquires a target page image containing learning objects; uses a pre-trained object recognition model to perform object recognition processing on the target page image to determine a first type of learning object; uses a pre-trained feature vector extraction model to perform feature extraction processing on the target page image to obtain a target feature vector and a second type of learning object; and determines the target type of learning object in the target page image based on the first type, the target feature vector, and the second type. Therefore, for page images containing multiple learning objects with small differences, two different models can be used to perform object recognition and feature extraction on the target page image. By combining the object recognition results and feature extraction results, the final category of the learning object can be determined, improving the recognition accuracy of the learning object. Even for page images of poor quality, and / or page images containing multiple different learning objects with small differences, the type of learning object can be accurately identified, and then an explanatory video matching the learning object can be loaded, thereby enhancing students' learning interest.

[0135] In some optional embodiments, the object recognition module 520 includes:

[0136] The first self-attention processing unit is used to perform self-attention processing on the target page image according to the target coefficient based on the self-attention network in the object recognition model to obtain the first self-attention feature.

[0137] The first classification unit is used to classify the first self-attention feature based on the classification network in the object recognition model to obtain the first type.

[0138] In some optional embodiments, the feature extraction module 530 includes:

[0139] The second self-attention processing unit is used to perform self-attention processing on the target page image according to the target coefficient based on the self-attention network in the feature vector extraction model to obtain the second self-attention feature.

[0140] The second classification unit is used to classify the second self-attention feature based on the classification network in the feature vector extraction model to obtain the target feature vector and the second type.

[0141] In some optional embodiments, the target coefficient is Y = f(W / (max(SUM(W)-W,W))*X+b), where Y is the output value of the neuron in the self-attention network, f(*) is the activation function, W is the weight of the connection between the neuron in the self-attention network and the neuron in the classification network, X is the input value of the neuron in the self-attention network, and b is the bias value.

[0142] In some optional embodiments, the type determination module 540 includes:

[0143] The search unit is used to search from a pre-built feature vector search library for multiple vectors to be traversed that have a similarity to the target feature vector that satisfies the first similarity condition.

[0144] The first determining unit is used to determine the target type based on the label type, the first type, and the second type carried by the plurality of vectors to be traversed, respectively.

[0145] In some optional embodiments, the first determining unit is specifically used for:

[0146] Determine whether the first type and the second type are consistent;

[0147] If the first type is the same as the second type, then the first type is taken as the target type;

[0148] If the first type is inconsistent with the second type, then the vector with the highest similarity is selected from the plurality of vectors to be traversed, and used as the target similarity variable of the target feature vector;

[0149] Determine whether the label type carried by the first type is consistent with that carried by the target similarity vector;

[0150] If the first type is consistent with the label type carried by the target similarity vector, then the first type is taken as the target type.

[0151] In some optional embodiments, the first determining unit is further configured to:

[0152] If the label type carried by the first type is inconsistent with that carried by the target similarity vector, then it is determined whether the label type carried by the target similarity variable is consistent with that carried by the second type.

[0153] If the label type carried by the target similarity variable is consistent with the second type, then the label type carried by the target similarity variable shall be taken as the target type.

[0154] In some optional embodiments, the first determining unit is further configured to:

[0155] If the label type carried by the target similarity variable is inconsistent with the second type, multiple first candidate feature vectors consistent with the label type carried by the target feature vector are searched from the multiple vectors to be traversed, and multiple second candidate feature vectors consistent with the first type are searched from the multiple vectors to be traversed.

[0156] Increase the similarity of each first candidate feature vector by a first value, and increase the similarity of each second candidate feature vector by a second value.

[0157] The vector to be traversed with the highest similarity is selected from the similarity between the first candidate feature vector after increasing the first value and the second candidate feature vector after increasing the second value, and the label type carried by the vector to be traversed with the highest similarity is taken as the target type.

[0158] In some alternative embodiments, the device further includes:

[0159] The first determining module is used to determine the first type as the target type.

[0160] In some alternative embodiments, the device further includes:

[0161] The first search module is used to search from a pre-built feature vector search library for multiple vectors to be traversed that have a similarity to the target feature vector that satisfies the first similarity condition.

[0162] The first selection module is used to select the vector with the highest similarity from the plurality of vectors to be traversed, and use it as the target similarity variable of the target feature vector;

[0163] The judgment module is used to determine whether the label type carried by the target similarity variable is consistent with the second type;

[0164] The second determining module is used to determine the target type if the label type carried by the target similar variable is consistent with the second type.

[0165] In some alternative embodiments, the device further includes:

[0166] The second search module is used to search for multiple first candidate feature vectors that are consistent with the label type carried by the target feature vector from the multiple vectors to be traversed if the label type carried by the target feature vector is inconsistent with the second type.

[0167] The similarity adjustment module is used to increase the first value of the similarity corresponding to each first candidate feature vector.

[0168] The second selection module is used to select the traversable vector with the highest similarity from the similarity corresponding to the first candidate feature vector after the first value is increased, and to take the label type carried by the traversable vector with the highest similarity as the target type.

[0169] In some optional embodiments, the image acquisition module 510 includes:

[0170] The image acquisition unit is used to acquire initial page images;

[0171] The detection unit is used to perform four-point bounding box detection on the initial page image using a pre-trained page detection model to obtain the page position information of the initial page image.

[0172] A cropping unit is used to crop candidate page images from the initial page image based on the page position information;

[0173] A correction unit is used to correct the initial shape of the pages in the candidate page image to the target shape, thereby obtaining the target page image.

[0174] In some optional embodiments, the target page image contains multiple learning objects, and the similarity between any two learning objects is greater than a preset similarity threshold; and / or,

[0175] The resolution of the target page image is less than a preset resolution threshold.

[0176] It should be noted that, Figure 5 The learning object recognition device 500 shown can perform... Figures 1 to 4 The various steps in the method embodiment shown are implemented. Figures 1 to 4 The processes and effects in the method embodiments shown are not described in detail here.

[0177] Exemplary embodiments of this disclosure also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform a method according to an embodiment of this disclosure.

[0178] Exemplary embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to embodiments of this disclosure.

[0179] Exemplary embodiments of this disclosure also provide a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this disclosure.

[0180] refer to Figure 6 The present invention describes a structural block diagram of an electronic device 600 that can serve as a server or client of the present disclosure. This is an example of a hardware device that can be applied to various aspects of the present disclosure, and the electronic device 600 can be the aforementioned electronic device. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0181] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0182] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, output unit 607, storage unit 608, and communication unit 609. Input unit 606 can be any type of device capable of inputting information to electronic device 600. Input unit 606 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 607 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 608 may include, but is not limited to, disks and optical discs. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0183] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the object recognition learning method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. In some embodiments, the computing unit 601 can be configured to perform the object recognition learning method by any other suitable means (e.g., by means of firmware).

[0184] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0185] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0186] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0187] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0188] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0189] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0190] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0191] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for recognizing learning objects, characterized in that, include: Obtain the image of the target page containing the learning object; Using a pre-trained object recognition model, object recognition processing is performed on the target book page image to determine the first type of the learned object; Using a pre-trained feature vector extraction model, feature extraction processing is performed on the target page image to obtain the target feature vector and the second type of the learning object; Based on the first type, the target feature vector, and the second type, the target type of the learning object in the target book page image is determined; The step of determining the target type of the learning object in the target book page image based on the first type, the target feature vector, and the second type includes: Search a pre-built feature vector search library for multiple vectors to be traversed that have a similarity to the target feature vector that satisfies the first similarity condition; Determine whether the first type and the second type are consistent; If the first type is the same as the second type, then the first type is taken as the target type; If the first type is inconsistent with the second type, then the vector with the highest similarity is selected from the plurality of vectors to be traversed, and used as the target similar vector of the target feature vector; Determine whether the label type carried by the first type is consistent with that carried by the target similarity vector; If the first type is consistent with the label type carried by the target similarity vector, then the first type is taken as the target type.

2. The method according to claim 1, characterized in that, The step of using a pre-trained object recognition model to perform object recognition processing on the target book page image and determine the first type of the learned object includes: Based on the self-attention network in the object recognition model, the target page image is processed with self-attention according to the target coefficient to obtain the first self-attention feature; The first self-attention feature is classified based on the classification network in the object recognition model to obtain the first type.

3. The method according to claim 1, characterized in that, The step of using a pre-trained feature vector extraction model to perform feature extraction processing on the target page image to obtain the target feature vector and the second type of the learning object includes: Based on the self-attention network in the feature vector extraction model, the target page image is subjected to self-attention processing according to the target coefficient to obtain the second self-attention feature. The second self-attention feature is classified based on the classification network in the feature vector extraction model to obtain the target feature vector and the second type.

4. The method according to claim 2 or 3, characterized in that, The target coefficient is Y = f(W / (max(SUM(W)-W,W))*X+b), where Y is the output value of the neuron in the self-attention network, f(*) is the activation function, W is the weight of the connection between the neuron in the self-attention network and the neuron in the classification network, X is the input value of the neuron in the self-attention network, and b is the bias value.

5. The method according to claim 1, characterized in that, Also includes: If the first type is inconsistent with the label type carried by the target similarity vector, then it is determined whether the label type carried by the target similarity vector is consistent with the second type. If the label type carried by the target similarity vector is consistent with the second type, then the label type carried by the target similarity vector shall be taken as the target type.

6. The method according to claim 5, characterized in that, Also includes: If the label type carried by the target similar vector is inconsistent with the second type, multiple first candidate feature vectors consistent with the label type carried by the target feature vector are searched from the multiple vectors to be traversed, and multiple second candidate feature vectors consistent with the first type are searched from the multiple vectors to be traversed. Increase the similarity of each first candidate feature vector by a first value, and increase the similarity of each second candidate feature vector by a second value. The vector to be traversed with the highest similarity is selected from the similarity between the first candidate feature vector after increasing the first value and the second candidate feature vector after increasing the second value, and the label type carried by the vector to be traversed with the highest similarity is taken as the target type.

7. The method according to claim 1, characterized in that, After performing object recognition processing on the target book page image using a pre-trained object recognition model to determine the first type of the learned object, the method further includes: The first type is taken as the target type.

8. The method according to claim 1, characterized in that, After performing feature extraction processing on the target page image using a pre-trained feature vector extraction model to obtain the target feature vector and the second type of the learning object, the method further includes: Search a pre-built feature vector search library for multiple vectors to be traversed that have a similarity to the target feature vector that satisfies the first similarity condition; Select the vector with the highest similarity from the plurality of vectors to be traversed, and use it as the target similarity vector of the target feature vector; Determine whether the label type carried by the target similarity vector is consistent with the second type; If the label type carried by the target similarity vector is consistent with the second type, then the label type carried by the target similarity vector shall be taken as the target type.

9. The method according to claim 8, characterized in that, Also includes: If the label type carried by the target similar vector is inconsistent with the second type, multiple first candidate feature vectors that are consistent with the label type carried by the target feature vector are searched from the multiple vectors to be traversed. Increase the similarity score of each first candidate feature vector by the first value. Select the vector to be traversed with the highest similarity from the first candidate feature vector after increasing the first value, and take the label type carried by the vector to be traversed with the highest similarity as the target type.

10. The method according to claim 1, characterized in that, The step of obtaining the target page image containing the learning object includes: Get the initial page image; Using a pre-trained page detection model, four-point bounding box detection is performed on the initial page image to obtain the page position information of the initial page image; Candidate page images are cropped from the initial page image based on the page position information; The initial shape of the page in the candidate page image is corrected to the target shape to obtain the target page image.

11. The method according to claim 1, characterized in that, The target page image contains multiple learning objects, and the similarity between any two learning objects is greater than a preset similarity threshold; and / or, The resolution of the target page image is less than a preset resolution threshold.

12. A learning object recognition device, characterized in that, include: The image acquisition module is used to acquire images of target book pages containing learning objects; The object recognition module is used to perform object recognition processing on the target book page image using a pre-trained object recognition model to determine the first type of the learned object; The feature extraction module is used to perform feature extraction processing on the target page image using a pre-trained feature vector extraction model to obtain the target feature vector and the second type of the learning object; A type determination module is used to determine the target type of the learning object in the target book page image based on the first type, the target feature vector, and the second type; The type determination module includes: The search unit is used to search from a pre-built feature vector search library for multiple vectors to be traversed that have a similarity to the target feature vector that satisfies the first similarity condition. The first determining unit is used to determine whether the first type and the second type are consistent; If the first type is the same as the second type, then the first type is taken as the target type; If the first type is inconsistent with the second type, then the vector with the highest similarity is selected from the plurality of vectors to be traversed, and used as the target similar vector of the target feature vector; Determine whether the label type carried by the first type is consistent with that carried by the target similarity vector; If the first type is consistent with the label type carried by the target similarity vector, then the first type is taken as the target type.

13. An electronic device, characterized in that, include: processor; Memory, used to store executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method of any one of claims 1-11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, The storage medium stores a computer program that, when executed by a processor, causes the processor to implement the method described in any one of claims 1-11.

Images