A method for scanning and searching questions, intelligent scanning pen and storage medium
By extracting and matching the image feature vectors of the scanned pictures, combining deep learning models and structured information, the accuracy and continuity of scanning questions in the existing technology are solved, the efficiency and accuracy of question retrieval is improved, and the user is improved Experience.
Patent Information
- Application Number
- CN202111123325.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-24
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-09-24
AI Technical Summary
The existing scanning and search methods rely on OCR to recognize image characters, which leads to users who need to maintain consistency when scanning test questions. Missing scans or rescans will affect the search results, and the accuracy is low and the user experience is poor.
By extracting the image feature vectors of the scanned pictures, using deep learning models for feature extraction and database matching, and searching with structured information of text, images, and symbols, reducing the scanning continuity requirements and improving retrieval efficiency and accuracy.
It improves the accuracy and user experience of scanning questions, reduces the difficulty of operation, realizes all-round question retrieval, and reduces users' requirements for scanning continuity.
Smart Images

Figure CN113821712B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of online education, and in particular to a method for scanning and searching questions, an intelligent scanning pen and a storage medium. Background Art
[0002] Online education can share high-quality educational resources through the Internet platform, which greatly facilitates users and reduces the cost of education. Photo search - a commonly used online education method, users can upload test questions by taking photos through smart devices (such as mobile phones). The cloud server will analyze and extract the uploaded test questions, and retrieve the test database to obtain relevant test search results, and return them to the user, which can greatly facilitate users to understand and master difficult test questions.
[0003] Compared with searching by taking photos, searching by scanning with the help of a smart scanning pen cannot obtain the picture of the entire test question by taking a photo. The only way to obtain the entire test question is to scan it line by line with the scanning camera of the smart scanning pen, and then upload and search the test question. This requires users to maintain continuity during the scanning process of the test question, and if there is a missed scan or repeated scan, it will affect the search results of the test question, resulting in a large gap in the search results of the test question. This is mainly because the existing scanning search mainly relies on OCR processing of the scanned image to obtain the image characters, and the test question is retrieved based on the characters. Therefore, the integrity and accuracy of the scanned image have a greater impact on the results of the test question search.
[0004] Therefore, how to reduce the difficulty of scanning and searching for questions for users, improve the accuracy of scanning and searching for questions, and thus improve the user experience is a technical problem to be solved by the present invention.
[0005] In view of this, the present invention is proposed. Summary of the invention
[0006] In order to solve the technical problem of how to reduce the difficulty of scanning and searching for questions by users, improve the accuracy of scanning and searching for questions, and enhance the user experience, the present invention provides a method for scanning and searching for questions, an intelligent scanning pen, and a storage medium. The specific technical scheme is as follows:
[0007] The present invention proposes a method for scanning and searching questions, comprising:
[0008] Get the scanned images during the scanning process of the test questions;
[0009] Extracting feature vectors for describing image features of scanned images;
[0010] According to the extracted feature vectors, search and match are performed in the test question database, and the search results are fed back.
[0011] As an optional implementation manner of the present invention, extracting a feature vector for describing image features of a scanned image includes:
[0012] A set of feature vectors of the scanned image is extracted through a deep learning model;
[0013] The feature vector is used to describe the structured information of text, images, and symbols in the scanned image.
[0014] As an optional implementation of the present invention, performing search and matching in the test question database according to the extracted feature vector and feeding back the search result includes:
[0015] Retrieve and match the structured information of the text, image, and symbol of each feature vector extracted with the structured information of the text, image, and symbol of the test question stored in the test question database;
[0016] The structured information of text, images and symbols in the retrieval matching results is sorted and weighted to obtain the test question search results.
[0017] As an optional implementation of the present invention, for science test questions, the weighted value of the structural information of symbols is greater than the weighted value of the structural information of text and images;
[0018] For liberal arts questions, the weighted value of text structured information is greater than that of images and symbols;
[0019] For graphic questions, the weighted value of the structural information of the image is greater than that of the structural information of the text and symbol;
[0020] Optionally, the subject to which the test question belongs is determined by extracting image features of the scanned image during the scanning process, or by receiving a test question subject setting instruction from the user before the scanning begins.
[0021] As an optional implementation of the present invention, performing search and matching in the test question database according to the extracted feature vector and feeding back the search result further comprises:
[0022] The test question search results are sorted and weighted based on the comprehensive search popularity and / or user historical search records, and the search results are fed back.
[0023] As an optional implementation manner of the present invention, the weighted operation of ranking the comprehensive search popularity of the test question search results includes:
[0024] Use big data to find popular questions in the test search results, and then perform weighted operations on the popular questions in the ranking;
[0025] The weighted operation of ranking the test question search results based on the historical search records of the user includes:
[0026] The test question search results are weighted in the sorting according to the grade and / or subject preference in the user's historical search records.
[0027] As an optional implementation manner of the present invention, the scanning picture obtained during the scanning test question process includes:
[0028] Get the scanned images in real time during the scanning process of the test questions;
[0029] Perform real-time processing on the scanned images to extract feature vectors, and perform real-time retrieval and matching in the test database based on the extracted feature vectors;
[0030] When the matching similarity of the feature vector reaches the set value, the search result is fed back.
[0031] As an optional implementation manner of the present invention, the scanning picture obtained during the scanning test question process includes:
[0032] Obtaining the scanning action and / or the length of the scanned image during the test question scanning process, wherein the scanning action includes scanning on and scanning off, and one scanning action includes one scanning on and one scanning off;
[0033] When the length of the scanned image of a scanning action reaches a length threshold, the scanned image of the scanning action is obtained and spliced with the scanned image of the previous scanning action to obtain a spliced image;
[0034] The spliced images are processed to extract feature vectors, and search and match are performed in the test database based on the extracted feature vectors to provide feedback on the search results.
[0035] The present invention also provides an intelligent scanning pen, including a scanning module and a control module:
[0036] The control module obtains the scanned images during the process of the scanning module scanning the test questions;
[0037] The control module extracts a feature vector for describing image features of the scanned image;
[0038] The control module uploads the extracted feature vector to the server, and the server searches and matches the test question database and feeds back the search results.
[0039] The present invention also provides a storage medium storing a computer executable program, which, when executed, implements the method of scanning and searching questions.
[0040] Compared with the prior art, the present invention has the following beneficial effects:
[0041] The scanning and searching method of the present invention realizes scanning and searching by extracting feature vectors for describing image features of scanned pictures, and performing search and matching in a test question database according to the extracted feature vectors. The feature vectors include a plurality of different image features that respectively describe the pictures, and each feature vector is searched and matched in the test question database respectively. The search result is fed back based on the search and matching results of each feature vector, thereby realizing a full-scale search of scanned pictures from multiple dimensions. Compared with a single method that relies on OCR to recognize picture characters for character matching, the efficiency and accuracy of test question retrieval are greatly improved.
[0042] In addition, since the scanning and searching method of the present invention comprehensively considers the image features such as information, structure and symbols of the scanned image, the user does not need to scan the entire test question before searching for the question, and can also obtain an extremely high test question retrieval accuracy rate; and the continuity requirements for test question scanning are also reduced, which greatly reduces the user's search operation requirements and improves the user's scanning and searching experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 An overall processing flow chart of the scanning and question searching method according to an embodiment of the present invention;
[0044] Figure 2 Flow chart of the scanning question search method according to the embodiment of the present invention Figure 1 ;
[0045] Figure 3 Flow chart of the scanning question search method according to the embodiment of the present invention Figure 2 ;
[0046] Figure 4 Flow chart of the scanning question search method according to the embodiment of the present invention Figure 3 ;
[0047] Figure 5 The hardware block diagram of the smart scanning pen according to the embodiment of the present invention;
[0048] Figure 6 A schematic structural diagram of an intelligent scanning pen according to an embodiment of the present invention.
[0049] Explanation of the numbers in the accompanying drawings: 1-main core CPU 2-battery power supply and communication interface module 3-eSIM card 4-LTE, WIFI / BT radio frequency module 5-EMCP memory module 6-touch display module 7-audio MIC and speaker module 8-fill light control module 9-button module 10-six-axis sensor 11-camera module 12-first camera 13-second camera 14-audio switch 15-power chip 16-communication interface 17-battery 18-WiFi / BT module 19-communication antenna 20-antenna chip 21-audio MIC 22-speaker 23-control switch 24-fill light 25-pen body 26-pen tip. DETAILED DESCRIPTION
[0050] To make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them.
[0051] Therefore, the following detailed description of the embodiments of the present invention is not intended to limit the scope of the invention claimed for protection, but merely represents some embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.
[0053] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0054] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the invention product is usually placed when in use, or the orientation or positional relationship commonly understood by those skilled in the art. Such terms are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.
[0055] See also Figure 1 As shown, this embodiment provides a method for scanning and searching questions, including:
[0056] Get the scanned images during the scanning process of the test questions;
[0057] Extracting feature vectors for describing image features of scanned images;
[0058] According to the extracted feature vectors, search and match are performed in the test question database, and the search results are fed back.
[0059] The scanning and searching method of the present embodiment realizes scanning and searching by extracting feature vectors for describing the image features of the scanned pictures, and performing search and matching in the test question database according to the extracted feature vectors. The feature vectors include a plurality of different image features that respectively describe the pictures, and each feature vector is searched and matched in the test question database respectively. The search results are fed back based on the search and matching results of each feature vector, thereby realizing a full-scale search of the scanned pictures from multiple dimensions. Compared with the method of solely relying on OCR to recognize the characters in the pictures for character matching, the efficiency and accuracy of the test question retrieval are greatly improved.
[0060] In addition, since the scanning and searching method of this embodiment comprehensively considers the image features such as information, structure and symbols of the scanned image, the user does not need to scan the entire test question before searching for the question, and can also obtain a very high test question retrieval accuracy rate; and the continuity requirements for test question scanning are also reduced, which greatly reduces the user's search operation requirements and improves the user's scanning and searching experience.
[0061] See also Figure 2 As shown, as an optional implementation of this embodiment, the extracting of a feature vector for describing the image features of the scanned image includes:
[0062] A set of feature vectors of the scanned image is extracted through a deep learning model;
[0063] The feature vector is used to describe the structured information of text, images, and symbols in the scanned image.
[0064] Image features mainly include color features, texture features, shape features and spatial relationship features of images. Color features are a global feature that describes the surface properties of the scene corresponding to an image or an image region; texture features are also a global feature that also describes the surface properties of the scene corresponding to an image or an image region; there are two types of representation methods for shape features, one is contour features and the other is regional features. The contour features of an image are mainly for the outer boundaries of an object, while the regional features of an image are related to the entire shape region; spatial relationship features refer to the spatial position or relative direction relationship between multiple targets segmented from an image. These relationships can also be divided into connection / adjacency relationships, overlap / overlapping relationships and inclusion / inclusion relationships. The image features extracted in this embodiment to describe the scanned image mainly include texture features, regional shape features and spatial relationship features.
[0065] The deep learning model of this embodiment is obtained by training based on input pictures and output pictures. The deep learning model extracts each feature vector in the scanned picture, performs semantic relevance matching through the feature vector of the structured information of the text in the picture, and performs similarity matching through the feature vector of the structured information of the image and symbol of the picture.
[0066] Further, see Figure 2 As shown, the search and matching is performed in the test question database according to the extracted feature vector, and the feedback search result includes:
[0067] Retrieve and match the structured information of the text, image, and symbol of each feature vector extracted with the structured information of the text, image, and symbol of the test question stored in the test question database;
[0068] The structured information of text, images and symbols in the retrieval matching results is sorted and weighted to obtain the test question search results.
[0069] The scanning and searching method of this embodiment matches the structured information of the text, image, and symbol of each feature vector with the test question database, which can greatly improve the search speed and accuracy. At the same time, the structured information of the text, image, and symbol is sorted and weighted for the search matching results, and sorting and weighting can be performed for different question types to further improve the accuracy of the search.
[0070] Specifically, for science questions, the weighted value of the structured information of symbols is greater than the weighted value of the structured information of text and images. The science questions of this embodiment include mathematics, chemistry, and physics. Specifically, the weighted values of the weighted operation can be set for the mathematical symbols of mathematics questions, the chemical symbols of chemistry questions, and the physical symbols of physics questions.
[0071] For liberal arts test questions, the weighted value of structured information of text is greater than the weighted value of structured information of images and symbols; the liberal arts test questions in this embodiment include Chinese, English, history, politics, etc.
[0072] For graphic test questions, the weighted value of the structured information of the image is greater than the weighted value of the structured information of the text and symbol; the graphic test questions of this embodiment include graphic recognition test questions for young children, etc.
[0073] Optionally, the subject to which the test question belongs can be determined by extracting image features of the scanned image during the scanning process. That is, the feature vector of the extracted scanned image can be used to determine whether it contains mathematical symbols, chemical symbols or physical symbols, thereby determining whether it belongs to mathematics, chemistry or physics.
[0074] Alternatively, the subject to which the test question belongs can be determined by receiving the test question subject setting instruction from the user before scanning begins. The user needs to set the subject to which the test question belongs before scanning the test question, and can switch between the built-in subject catalog by pressing buttons or scrolling the wheel.
[0075] As an optional implementation of this embodiment, see Figure 2 As shown, in a scanning and searching method of this embodiment, searching and matching are performed in a test question database according to the extracted feature vectors, and feeding back the search result further includes:
[0076] The test question search results are sorted and weighted based on the comprehensive search popularity and / or user historical search records, and the search results are fed back.
[0077] In the scanning and searching method of the present embodiment, the search process also comprehensively considers the search popularity and / or the user's historical search records to perform weighted sorting operations. Questions with high search popularity have higher weighted values and are ranked higher. The user's historical search behavior helps to preliminarily locate the user, so that questions that are more suitable for the user's current stage are ranked higher and displayed, thereby improving the accuracy of scanning and searching.
[0078] Specifically, the weighted ranking operation for the comprehensive search popularity of the test question search results described in this embodiment includes:
[0079] Use big data to find popular questions in the test search results, and then perform weighted operations on the popular questions in the sorting.
[0080] In this embodiment, the weighted operation of ranking the test question search results based on the user's historical search records includes:
[0081] The test question search results are weighted in the sorting according to the grade and / or subject preference in the user's historical search records.
[0082] As an optional implementation of this embodiment, see Figure 3 As shown, the scanning image obtained in the scanning question search method of this embodiment includes:
[0083] Get the scanned images in real time during the scanning process of the test questions;
[0084] Perform real-time processing on the scanned images to extract feature vectors, and perform real-time retrieval and matching in the test database based on the extracted feature vectors;
[0085] When the matching similarity of the feature vector reaches the set value, the search result is fed back.
[0086] As an optional implementation of this embodiment, see Figure 4As shown, the scanning image obtained in the scanning question search method of this embodiment includes:
[0087] Obtaining the scanning action and / or the length of the scanned image during the test question scanning process, wherein the scanning action includes scanning on and scanning off, and one scanning action includes one scanning on and one scanning off;
[0088] When the length of the scanned image of a scanning action reaches a length threshold, the scanned image of the scanning action is obtained and spliced with the scanned image of the previous scanning action to obtain a spliced image;
[0089] The spliced images are processed to extract feature vectors, and search and match are performed in the test database based on the extracted feature vectors to provide feedback on the search results.
[0090] During a scan, if the length of the scanned image is lower than the minimum length threshold, the scanned image will be discarded and a message will be displayed indicating that the scan is invalid and you need to scan again. The scanned image will only be saved and spliced if its length reaches the length threshold.
[0091] The scanning and question searching method of this embodiment realizes real-time question searching during the process of scanning test questions. The user can choose whether to continue scanning based on the real-time question searching results. If the corresponding test questions are found during the scanning process, the user can choose to stop scanning, thereby improving the efficiency of scanning and question searching.
[0092] This embodiment also provides a smart scanning pen, including a scanning module and a control module:
[0093] The control module obtains the scanned images during the process of the scanning module scanning the test questions;
[0094] The control module extracts a feature vector for describing image features of the scanned image;
[0095] The control module uploads the extracted feature vector to the server, and the server searches and matches the test question database and feeds back the search results.
[0096] The smart scanning pen of this embodiment realizes scanning and question searching by extracting feature vectors used to describe the image features of the scanned pictures, and performing retrieval and matching in the test question database according to the extracted feature vectors. The feature vectors include multiple different image features that respectively describe the pictures. Each feature vector is searched and matched in the test question database respectively, and the search results are fed back based on the retrieval and matching results of each feature vector, thereby realizing a full-scale retrieval of the scanned pictures from multiple dimensions. Compared with the method of relying solely on OCR to recognize the characters in the pictures for character matching, the efficiency and accuracy of the test question retrieval are greatly improved.
[0097] In addition, since the smart scanning pen of this embodiment comprehensively considers the image features such as information, structure and symbols of the scanned image, the user does not need to scan the entire test question before searching for the test question, and can also obtain a very high test question retrieval accuracy rate; and the continuity requirements for test question scanning are also reduced, which greatly reduces the user's search operation requirements and improves the user's scanning and search experience.
[0098] As an optional implementation of this embodiment, the control module includes:
[0099] The image feature extraction unit extracts a set of feature vectors of the scanned image through a deep learning model; the feature vectors are used to describe the structured information of the text, images, and symbols of the scanned image.
[0100] Image features mainly include color features, texture features, shape features and spatial relationship features of images. Color features are a global feature that describes the surface properties of the scene corresponding to an image or an image region; texture features are also a global feature that also describes the surface properties of the scene corresponding to an image or an image region; there are two types of representation methods for shape features, one is contour features and the other is regional features. The contour features of an image are mainly for the outer boundaries of an object, while the regional features of an image are related to the entire shape region; spatial relationship features refer to the spatial position or relative direction relationship between multiple targets segmented from an image. These relationships can also be divided into connection / adjacency relationships, overlap / overlapping relationships and inclusion / inclusion relationships. The image features extracted in this embodiment to describe the scanned image mainly include texture features, regional shape features and spatial relationship features.
[0101] The deep learning model of this embodiment is obtained by training based on input pictures and output pictures. The deep learning model extracts each feature vector in the scanned picture, performs semantic relevance matching through the feature vector of the structured information of the text in the picture, and performs similarity matching through the feature vector of the structured information of the image and symbol of the picture.
[0102] Furthermore, the server searches and matches the test question database based on the extracted feature vectors, and the feedback search results include:
[0103] The server searches and matches the structured information of the text, image, and symbol of each feature vector respectively with the structured information of the text, image, and symbol of the test question stored in the test question database;
[0104] The structured information of text, images and symbols in the retrieval matching results is sorted and weighted to obtain the test question search results.
[0105] The server of this embodiment matches the structured information of the text, image, and symbol of each feature vector with the test question database, which can greatly improve the search speed and accuracy. At the same time, the structured information of the text, image, and symbol is sorted and weighted for the search matching results, and sorting and weighting can be performed for different question types to further improve the accuracy of the search.
[0106] Specifically, for science questions, the weighted value of the structured information of symbols is greater than the weighted value of the structured information of text and images. The science questions of this embodiment include mathematics, chemistry, and physics. Specifically, the weighted values of the weighted operation can be set for the mathematical symbols of mathematics questions, the chemical symbols of chemistry questions, and the physical symbols of physics questions.
[0107] For liberal arts test questions, the weighted value of structured information of text is greater than the weighted value of structured information of images and symbols; the liberal arts test questions in this embodiment include Chinese, English, history, politics, etc.
[0108] For graphic test questions, the weighted value of the structured information of the image is greater than the weighted value of the structured information of the text and symbol; the graphic test questions of this embodiment include graphic recognition test questions for young children, etc.
[0109] Optionally, the server determines the subject to which the test question belongs by extracting image features of the scanned image during the scanning process, that is, it determines whether the extracted feature vector of the scanned image contains mathematical symbols, chemical symbols or physical symbols, and determines whether it belongs to mathematics, chemistry or physics.
[0110] Alternatively, the server determines the subject to which the test question belongs by receiving the user's test question subject setting instruction before scanning begins. The user needs to set the subject to which the test question belongs before scanning the test question, and can switch between the built-in subject catalog by pressing buttons or scrolling the wheel.
[0111] As an optional implementation of this embodiment, the server of this embodiment performs search and matching in the test question database according to the extracted feature vector, and the feedback of the search result also includes:
[0112] The test question search results are sorted and weighted based on the comprehensive search popularity and / or user historical search records, and the search results are fed back.
[0113] During the question search process, the smart scanning pen in this embodiment also comprehensively considers the search popularity and / or the user's historical search records to perform weighted sorting operations. Questions with high search popularity have higher weighted values and are ranked higher. The user's historical search behavior helps to initially locate the user, so that questions that are more suitable for the user's current stage are ranked higher and displayed, thereby improving the accuracy of scanning and searching questions.
[0114] Specifically, the weighted ranking operation for the comprehensive search popularity of the test question search results described in this embodiment includes:
[0115] Use big data to find popular questions in the test search results, and then perform weighted operations on the popular questions in the sorting.
[0116] In this embodiment, the weighted operation of ranking the test question search results based on the user's historical search records includes:
[0117] The test question search results are weighted in the sorting according to the grade and / or subject preference in the user's historical search records.
[0118] As an optional implementation of this embodiment, the intelligent scanning pen of this embodiment obtains the scanned image in the process of scanning the test questions, including:
[0119] Get the scanned images in real time during the scanning process of the test questions;
[0120] Perform real-time processing on the scanned images to extract feature vectors, and perform real-time retrieval and matching in the test database based on the extracted feature vectors;
[0121] When the matching similarity of the feature vector reaches the set value, the search result is fed back.
[0122] As an optional implementation of this embodiment, the intelligent scanning pen of this embodiment obtains the scanned image in the process of scanning the test questions, including:
[0123] Obtaining the scanning action and / or the length of the scanned image during the test question scanning process, wherein the scanning action includes scanning on and scanning off, and one scanning action includes one scanning on and one scanning off;
[0124] When the length of the scanned image of a scanning action reaches a length threshold, the scanned image of the scanning action is obtained and spliced with the scanned image of the previous scanning action to obtain a spliced image;
[0125] The spliced images are processed to extract feature vectors, and search and match are performed in the test database based on the extracted feature vectors to provide feedback on the search results.
[0126] The smart scanning pen of this embodiment realizes real-time question searching during the process of scanning test questions. The user can choose whether to continue scanning based on the real-time question searching results. If the corresponding test questions are found during the scanning process, the user can choose to stop scanning, thereby improving the efficiency of scanning and searching.
[0127] As an optional implementation of this embodiment, see Figure 5 and Figure 6 As shown, the smart scanning pen of this embodiment includes:
[0128] Pen body 25;
[0129] The main core CPU1 is arranged in the pen body 25;
[0130] And the camera module 11 includes a first camera 12 and a second camera 13, the first camera 12 and the second camera 13 are respectively arranged at different positions of the pen body 25, and at least one camera is arranged at the position of the pen head 26 of the pen body 25, and the first camera 12 and the second camera 13 are respectively communicated with the main core CPU1.
[0131] The main core CPU1 of the smart scanning pen of this embodiment is the control center of the whole machine, which is used to send, receive and process data of the peripheral interface. The smart scanning pen of this embodiment includes a first camera 12 and a second camera 13, which can be applied to image acquisition needs in different scenarios, realize scanning, taking pictures and other functions, enrich the use scenarios of the smart pen, and improve the user experience.
[0132] As an optional implementation of this embodiment, the first camera 12 described in this embodiment is a scanning camera for scanning and searching, and the scanning camera is arranged at the position of the pen tip 26 of the pen body 25.
[0133] The first camera 12 of this embodiment is a scanning camera, which can scan text to be translated or homework questions to be analyzed, etc. After scanning, the image is transmitted to the CPU through the MIPI interface.
[0134] Furthermore, the second camera 13 described in this embodiment is a photo search camera used for photo search, and the photo search camera is arranged on the side wall of the upper part of the pen body 25.
[0135] The second camera 13 of this embodiment is a photo-search camera, which can take pictures of the homework questions to be analyzed and transmit the pictures to the CPU through the MIPI interface.
[0136] Therefore, the smart scanning pen of this embodiment includes a scanning search camera and a photo search camera, which provide a hardware basis for separately realizing scanning search and photo search, and can realize various learning functions such as dictionary translation, scanning questions, and taking photos of questions, which greatly meets the user's learning needs, improves the user's usage experience, and has a high product cost performance.
[0137] As an optional implementation of this embodiment, the smart scanning pen of this embodiment includes a fill light 24 and a control switch 23. The fill light 24 is installed on the pen head 26 and is close to the scanning camera position. The control switch 23 is installed on the pen body 25. The control switch 23 and the fill light 24 are respectively communicated with the main core CPU1, and the control switch 23 is used to control the opening and closing of the fill light 24.
[0138] The CPU of the smart scanner pen of this embodiment turns on and off the fill light 24 by controlling the switch information of the switch 23 . The fill light 24 is used to fill light for the scanning camera in the camera module 11 .
[0139] The control switch 23 and the fill light 24 of this embodiment together constitute a fill light control module 8 .
[0140] Optionally, the control switch 23 is a travel switch, which can generate travel information, and the CPU uses the travel information to turn on and off the fill light 24. The control switch 23 can also be replaced by a pressure sensor, but the overall hardware solution is the same.
[0141] As an optional implementation of this embodiment, the smart scanning pen includes a cellular mobile communication module, which is communicatively connected to the main core CPU and is used to access a cellular mobile communication network.
[0142] The smart scanning pen of this embodiment includes a cellular mobile communication module to enable joining the communication network, and can still meet the needs of online use in an environment without WiFi.
[0143] Furthermore, the cellular mobile communication module described in this embodiment includes an eSIM card 3 arranged in the pen body 25, and the eSIM card 3 is communicatively connected with the main core CPU1.
[0144] Or the cellular mobile communication module includes a SIM card slot for installing a SIM card, the SIM card slot is installed in the pen body in a removable / insertable manner, and the SIM card slot is communicatively connected to the main core CPU.
[0145] The eSIM card 3 of this embodiment or the SIM card inserted into the SIM card slot can access China Mobile / China Unicom / China Telecom communications and realize networking needs through mobile data.
[0146] The cellular mobile communication module described in this embodiment includes a communication antenna 19 and an antenna chip 20 that is communicatively connected to the communication antenna 19. The communication antenna 19 extends out of the pen body 25, and the antenna chip 20 is arranged in the pen body 25 and is communicatively connected to the main core CPU1.
[0147] The communication antenna 19 of this embodiment can adopt an LTE antenna, and the antenna chip 20 can adopt an LTE transceiver chip to realize the wireless communication data transmission and reception of the existing LTE.
[0148] The smart scanning pen of this embodiment includes a WiFi / BT module 18, which is connected to the main core CPU1 to realize wireless communication data transmission and reception of WIFI / BT.
[0149] In this embodiment, the LTE antenna, LTE transceiver chip and WiFi / BT module 18 are integrated into an LTE, WIFI / BT radio frequency module 4 .
[0150] The smart scanning pen of this embodiment integrates a scanning camera, a taking a photo camera, LTE, and WIFI / BT radio frequency modules, and can realize a variety of learning functions such as dictionary translation, scanning questions, and taking photos of questions locally and in the cloud. It is a smart pen product currently not available on the market, enriching the types of smart pen products and greatly improving students' learning experience.
[0151] As an optional implementation of this embodiment, the smart scanning pen described in this embodiment includes a six-axis sensor 10, and the six-axis sensor 10 is communicatively connected to the main core CPU1; the six-axis sensor 10 collects the posture / acceleration information of the smart pen and transmits it to the main core CPU1, and the main core CPU1 uses the information to remind the user to adjust the pen body angle.
[0152] As an optional implementation of this embodiment, the smart scanning pen described in this embodiment includes a battery module and a communication interface module 16, the battery module includes a battery 17 and a power chip 15, the power chip 15 is communicatively connected to the main core CPU1, and the battery 17 and the communication interface module 16 are respectively communicatively connected to the power chip 15; the battery 17, the power chip 15 and the communication interface module 16 constitute a charging and discharging circuit.
[0153] Optionally, the communication interface module 16 of this embodiment adopts a Type-C interface.
[0154] As an optional implementation of this embodiment, the smart scanning pen described in this embodiment includes an audio switch 14, and the audio switch 14, the power chip 15 and the communication interface 16 are communicatively connected with each other to form a USB / Audio communication circuit.
[0155] The battery 17 , the power chip 15 , the communication interface module 16 and the audio switch 14 of this embodiment together constitute a battery power supply and communication interface module 2 .
[0156] As an optional implementation of this embodiment, the smart scanning pen described in this embodiment includes an EMCP memory module 5. The EMCP memory module 5 is composed of EMMC and LPDDR3, and can realize data reading, writing and storage.
[0157] As an optional implementation of this embodiment, the smart scanning pen described in this embodiment includes a touch display module 6, and a touch screen can be selected. The screen is used to display the content, and the touch is used to realize touch interaction of the screen.
[0158] As an optional implementation of this embodiment, the smart scanning pen described in this embodiment includes an audio MIC and a speaker module 7, the audio MIC 21 is used to pick up voice information of external users, and the speaker 22 is used to play audio.
[0159] As an optional implementation scheme of this embodiment, the smart scanner pen described in this embodiment includes a button module 9, which includes a power button / lock screen button, a voice / Home button, and a camera button. The power button / lock screen button is long pressed to turn on and off, and a short press in the on state is to lock the screen. The voice / Home button is long pressed for voice and short pressed for Home, and the camera button is short pressed to take pictures.
[0160] The smart pen designed based on the present invention has a built-in dual camera, which can realize both dictionary translation and homework analysis functions; and because of the built-in LTE, it can not only be used at home, but also supports a variety of learning scenarios that cannot be covered by WIFI. It completely solves the limitations of the mainstream learning pens on the market with single functions and limited scenarios.
[0161] This embodiment also provides a storage medium storing a computer executable program, which, when executed, implements the method of scanning and searching for questions.
[0162] The storage medium described in the present embodiment may include a data signal propagated in a baseband or as a part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.
[0163] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the method of scanning and searching for questions.
[0164] The electronic device is presented in the form of a general-purpose computing device. The processor may be one or more than one and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.
[0165] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.
[0166] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).
[0167] It should be understood that the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.
[0168] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and can also be distributed and stored on the network, as long as it can enable the electronic device to execute the method according to the present invention.
[0169] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described in the present invention. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the above specific implementation methods. Therefore, any modification or equivalent replacement of the present invention; and all technical solutions and improvements thereof that do not depart from the spirit and scope of the invention are included in the scope of the claims of the present invention.
Claims
1. A method for scanning and searching questions, characterized in that: include: Get the scanned images during the scanning process of the test questions; Extracting feature vectors for describing image features of scanned images; Search and match the test question database based on the extracted feature vectors, and feedback the search results; The extracting of a feature vector for describing the image features of the scanned image comprises: A set of feature vectors of the scanned image is extracted through a deep learning model; The feature vector is used to describe the structured information of the text, image, and symbol of the scanned image; The search and matching is performed in the test question database according to the extracted feature vector, and the feedback of the search result includes: Retrieve and match the structured information of the text, image, and symbol of each feature vector extracted with the structured information of the text, image, and symbol of the test question stored in the test question database; The structured information of text, images and symbols is sorted and weighted for the search results to obtain the test question search results; For science questions, the weighted value of symbolic structured information is greater than that of text and image structured information; For liberal arts questions, the weighted value of text structured information is greater than that of images and symbols; For graphic questions, the weighted value of the structural information of the image is greater than that of the structural information of the text and symbol; Determine the subject to which the test question belongs by extracting image features of the scanned image during the scanning process, or determine the subject to which the test question belongs by receiving a test question subject setting instruction from the user before the scanning starts; The searching and matching in the test question database according to the extracted feature vector and feeding back the search result also include: Perform a weighted ranking operation on the comprehensive search popularity of the test question search results and / or the user's historical search records, and provide feedback on the search results; The weighted operation of ranking the comprehensive search heat of the test question search results includes: Use big data to find popular questions in the test search results, and then perform weighted operations on the popular questions in the ranking; The weighted operation of ranking the test question search results based on the historical search records of the user includes: For the test question search results, weighted calculation is performed in the sorting according to the grade and / or subject preference in the user's historical search records; The scanning pictures in the process of obtaining the scanning test questions include: Get the scanned images in real time during the scanning process of the test questions; Perform real-time processing on the scanned images to extract feature vectors, and perform real-time retrieval and matching in the test database based on the extracted feature vectors; When the matching similarity of the feature vector reaches the set value, the search result is fed back.
2. A scanning and searching method according to claim 1, characterized in that: The scanning pictures in the process of obtaining the scanning test questions include: Obtaining the scanning action and / or the length of the scanned image during the test question scanning process, wherein the scanning action includes scanning on and scanning off, and one scanning action includes one scanning on and one scanning off; When the length of the scanned image of a scanning action reaches a length threshold, the scanned image of the scanning action is obtained and spliced with the scanned image of the previous scanning action to obtain a spliced image; The spliced images are processed to extract feature vectors, and search and match are performed in the test database based on the extracted feature vectors to provide feedback on the search results.
3. A smart scanning pen, characterized in that: Including scanning module and control module: The control module obtains the scanned images during the process of the scanning module scanning the test questions; The control module extracts a feature vector for describing image features of the scanned image; The control module uploads the extracted feature vector to the server, and the server searches and matches the test question database and feeds back the search results. The control module comprises: An image feature extraction unit extracts a set of feature vectors of the scanned image through a deep learning model; the feature vectors are used to describe the structured information of the text, image, and symbol of the scanned image; The server searches and matches the test question database based on the extracted feature vectors, and the feedback search results include: The server searches and matches the structured information of the text, image, and symbol of each feature vector respectively with the structured information of the text, image, and symbol of the test question stored in the test question database; The structured information of text, images and symbols is sorted and weighted for the search results to obtain the test question search results; For science questions, the weighted value of symbolic structured information is greater than that of text and image structured information; For liberal arts questions, the weighted value of text structured information is greater than that of images and symbols; For graphic questions, the weighted value of the structural information of the image is greater than that of the structural information of the text and symbol; The server determines the subject to which the test question belongs by extracting image features of the scanned image during the scanning process, or the server determines the subject to which the test question belongs by receiving a test question subject setting instruction from the user before the scanning starts; The server searches and matches the test question database based on the extracted feature vectors, and the feedback search results also include: Perform a weighted ranking operation on the comprehensive search popularity of the test question search results and / or the user's historical search records, and provide feedback on the search results; The weighted operation of ranking the comprehensive search heat of the test question search results includes: Use big data to find popular questions in the test search results, and then perform weighted operations on the popular questions in the ranking; The weighted operation of ranking the test question search results based on the historical search records of the user includes: For the test question search results, weighted calculation is performed in the sorting according to the grade and / or subject preference in the user's historical search records; The scanned images obtained by the smart scanning pen during the scanning test include: Get the scanned images in real time during the scanning process of the test questions; Perform real-time processing on the scanned images to extract feature vectors, and perform real-time retrieval and matching in the test database based on the extracted feature vectors; When the matching similarity of the feature vector reaches the set value, the search result is fed back.
4. A storage medium, characterized in that: A computer executable program is stored, and when the computer executable program is executed, a method for scanning and searching questions as described in any one of claims 1-2 is implemented.
