Point reading method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to the technical field of computer processing, in particular to a pointing reading method and device, electronic equipment and storage medium, the method comprising: acquiring a current query image; performing finger recognition on the current query image to determine a pointing reading finger quantity, to obtain a current query mode corresponding to the pointing reading finger quantity, the pointing reading finger quantity being a quantity of fingers pointing to target content in the current query image; querying the target content according to the current query mode, and outputting a query result. The method directly determines the current query mode by using the pointing reading finger quantity in the query process, automatically adjusts the current query mode, and does not need to manually switch the current query mode, thereby improving the pointing reading efficiency.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer processing, and in particular, to a pointing reading method and device, an electronic device, and a storage medium. BACKGROUND

[0002] For learning hardware devices, pointing reading is a frequently used function for students to query Chinese and English words. In addition to Chinese and English words, Chinese and English sentences and whole paragraphs are also commonly queried. When pointing reading, the user needs to manually select the corresponding query mode in advance, such as querying words, sentences, or paragraphs, to perform the pointing reading operation. If the user wants to switch the query mode, the user needs to stop pointing reading and manually switch the query mode. SUMMARY

[0003] Therefore, the present disclosure provides a pointing reading method, device, electronic device, and storage medium to solve the problem of low pointing reading efficiency caused by manually switching the query mode.

[0004] According to a first aspect, the present disclosure provides a pointing reading method, comprising:

[0005] obtaining a current query image;

[0006] performing finger recognition on the current query image to determine a pointing reading finger quantity, to obtain a current query mode corresponding to the hand gesture, the pointing reading finger quantity being the number of fingers pointing to target content in the current query image;

[0007] querying the target content according to the current query mode, and outputting a query result.

[0008] According to a second aspect, the present disclosure also provides a pointing reading device, comprising:

[0009] a obtaining module configured to obtain a current query image;

[0010] a recognition module configured to perform finger recognition on the current query image to determine a pointing reading finger quantity, to obtain a current query mode corresponding to the pointing reading finger quantity, the pointing reading finger quantity being the number of fingers pointing to target content in the current query image;

[0011] a querying module configured to query the target content according to the current query mode, and output a query result.

[0012] According to a third aspect, the embodiments of the present disclosure provide an electronic device, comprising a memory and a processor, which are connected in communication with each other, the memory stores computer instructions, and the processor executes the point reading method according to the first aspect or any one of the embodiments of the first aspect by executing the computer instructions.

[0013] According to a fourth aspect, the embodiments of the present disclosure provide a computer readable storage medium, which stores computer instructions for causing the computer to execute the point reading method according to the first aspect or any one of the embodiments of the first aspect.

[0014] The point reading method provided by the embodiments of the present disclosure can determine the current query mode corresponding to the point reading finger quantity in the point reading process, query the content of the current query image by using the current query mode, directly determine the current query mode by using the point reading finger quantity in the query process, automatically adjust the current query mode, and does not need to manually switch the current query mode, thereby improving the point reading efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure or the prior art, the drawings needed in the description of the embodiments or the prior art will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present disclosure, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0016] Figure 1 is a flowchart of the point reading method according to the embodiments of the present disclosure;

[0017] Figure 2 is a flowchart of the point reading method according to the embodiments of the present disclosure;

[0018] Figure 3 is a flowchart of the point reading method according to the embodiments of the present disclosure;

[0019] Figures 4a-4c is a schematic diagram of the application scenario of the point reading method according to the embodiments of the present disclosure;

[0020] Figure 5 is a structural block diagram of the point reading device according to the embodiments of the present disclosure;

[0021] Figure 6 is a hardware structure schematic diagram of the electronic device provided by the embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0023] The commonly used point-and-read method involves interacting with the device to first determine the query mode. During the reading process, the user moves their finger over the content to be queried, and the system captures an image of the finger's position to determine the query content. The system then performs the query and outputs the result. For example, if the current query mode is for searching a character, the system will play the corresponding character's audio or provide a translation. To switch from searching for a character to searching for a sentence, the user must stop reading, interact with the device again to change the query mode, and then continue reading for sentence queries.

[0024] This method requires the user to stop interacting with the reading device to switch query modes. Furthermore, when searching for Chinese words, since Chinese words typically contain at least two characters, users need to quickly move their finger to select the item being searched. For example, when searching for a sentence, the user needs to quickly slide their finger under the sentence to select it. However, this rapid finger-sliding method may lead to some characters being missed in the Chinese vocabulary, resulting in a lower recognition rate.

[0025] Based on this, a point-and-read method is proposed in this embodiment. During the point-and-read process, the query mode is switched by adjusting the number of fingers used for pointing, without affecting the current point-and-read; that is, the query mode is switched while the point-and-read is being performed. The electronic device executing this point-and-read method can be a smart lamp, tablet, smart speaker, etc., and there are no specific restrictions on its form, as long as the electronic device has image acquisition capabilities and can execute the point-and-read method of this embodiment.

[0026] According to an embodiment of this disclosure, a point-reading method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0027] This embodiment provides a point-and-read method that can be used in the aforementioned electronic device. Figure 1 This is a flowchart of a point-reading method according to an embodiment of the present disclosure, such as... Figure 1 As shown, the process includes the following steps:

[0028] S11, retrieve the currently queried image.

[0029] During the reading process, the electronic device collects the currently queried image in real time. The query cycle for collecting the currently queried image can be once every 0.5 seconds, once every 0.1 seconds, etc. The specific query cycle can be set according to actual needs, and no limit is made here.

[0030] The current query image is an image that includes a finger pointing and the query content. The query mode and query content are determined simultaneously by pointing the finger. The number of fingers pointing determines the current query mode, and the content pointed to by the finger determines the target content of the query. Thus, the query mode can be switched without introducing other operations.

[0031] S12, perform finger recognition on the current query image to determine the number of fingers pointing, so as to obtain the current query mode corresponding to the number of fingers pointing.

[0032] The number of fingers pointing refers to the number of fingers pointing at the target content in the currently queried image.

[0033] After activating the point-and-read function, if the currently queried image is the first image captured, the electronic device identifies the number of fingers pointing at it to determine the current query mode. If the currently queried image is the second or subsequent image, the electronic device identifies the number of fingers pointing at it to determine if this number has changed compared to the number of fingers pointing at the previous queried image. If there is no change, the query mode corresponding to the previous queried image is used directly; if there is a change, the query mode needs to be switched to obtain the current query mode corresponding to the number of fingers pointing at the current queried image. The current query mode includes, but is not limited to, searching for characters, sentences, or paragraphs, with the target content being characters, sentences, or paragraphs respectively.

[0034] When an electronic device identifies the number of fingers pointing in a query image, it first determines the area covered by the hand within the image, then performs feature analysis on that area to determine the number of fingers. For example, by collecting multiple sample images of fingers used for pointing, a finger-counting recognition model can be trained using these images. In application, the identified coverage area is input into the model to output the number of fingers. Alternatively, since the positional relationships between the fingernails differ depending on the number of fingers pointing, the electronic device can determine the number of fingers by recognizing the fingernails in the query image. The specific method for identifying the number of fingers is not limited here; it can be set according to actual needs.

[0035] For example, when an electronic device identifies the number of fingers being used for reading, it first segments the fingers from the current query image, and then filters out the number of fingers pointing towards content in the current query image. If a finger is bent and pointing towards the palm, the fingernail cannot be identified; however, if the finger points towards content in the current query image, the fingernail can be identified. Therefore, the electronic device can determine the number of fingers being used for reading by identifying the fingernails from the segmented fingers. Alternatively, the electronic device can identify the fingertips in the current query image to determine the number of fingers being used for reading.

[0036] The electronic device maintains a correspondence between the number of fingers pointing at the screen and the query mode. After identifying the number of fingers, this correspondence is queried to determine the current query mode corresponding to that number of fingers. Different numbers of fingers indicate different query modes, and since the fingers are used for pointing, instant querying is possible.

[0037] It should be noted that the number of fingers used for pointing in this embodiment refers to the number of fingers pointing at the content of the currently queried image, not the number of fingers hovering in the air. Therefore, in some optional embodiments, the positional relationship between the fingers and the currently queried image needs to be determined before determining the number of fingers. If a finger is pressed on the currently queried image, the number of fingers for pointing is identified; if a finger is not pressed on the currently queried image, the number of fingers for pointing is not identified, indicating that the user has not started pointing, and a pointing reminder can be output, etc.

[0038] Determining the positional relationship between a finger and the current query image can be achieved by analyzing the current query image to determine the positional relationship between the finger's shadow and the finger itself; alternatively, it can involve identifying the area of ​​the fingernail, comparing the size of the fingernail area with the text in the current query image, and thus determining the positional relationship between the finger and the current query image. This is because the size of the fingernail corresponds to a certain threshold range, and the size of the text can also be determined. If a finger is pressed on the current query image, the size relationship between the fingernail and the nearest text fingertip can be determined.

[0039] The specifics of this step will be described in detail below.

[0040] S13, perform a query on the target content according to the current query mode and output the query results.

[0041] The current query mode determines the user's current query, such as searching for a word, sentence, or paragraph. After the current query mode is determined, the electronic device analyzes the current query image, extracts the target content corresponding to the current query mode, performs the query, and outputs the results. The target content corresponding to the current query mode can be what the user is pointing to, or any content within the current query image, etc.

[0042] Specifically, the electronic device can perform queries in either offline or online mode. In offline mode, the electronic device has a built-in knowledge base. It matches the extracted query content against the knowledge base and outputs the query results. In online mode, the electronic device sends the extracted content to a backend server, which then performs the query, obtains the results, and sends them back to the electronic device for output. The electronic device can support either offline or online mode, or both. If both are supported, the electronic device prioritizes online mode. If it detects poor network conditions, it switches to offline mode.

[0043] When electronic devices output search results, they employ methods including, but not limited to, voice output and image output. Taking Chinese vocabulary search as an example, due to the diverse ways Chinese words can be formed, words composed of three or more characters can be broken down into multiple words. Taking "disagree" as an example, in vocabulary search mode, after recognizing "disagree," the electronic device outputs results based on semantic priority. That is, the electronic device maintains a usage priority for each word, prioritizing the output of words with higher priority. The output words include "different," "agree," and "disagree," and all words are output according to priority. Furthermore, the electronic device provides an interactive selection method, allowing users to choose from all the output words. For example, the electronic device displays these three words on the interface for the user to choose from; when the user selects a word, the corresponding search results are output.

[0044] The specifics of this step will be described in detail below.

[0045] The point-reading method provided in this embodiment performs finger recognition on the current query image during the point-reading process to obtain the number of fingers used for point-reading, determines the current query mode corresponding to the number of fingers used for point-reading, and uses the current query mode to query the content of the current query image. This realizes that during the query process, the current query mode is directly determined by the number of fingers used for point-reading, and the current query mode is automatically adjusted without the need for manual switching of the current query mode, thus improving point-reading efficiency.

[0046] This embodiment provides a point-and-read method that can be used in the aforementioned electronic devices, such as mobile phones, tablets, and desk lamps. Figure 2 This is a flowchart of a point-reading method according to an embodiment of the present disclosure, such as... Figure 2 As shown, the process includes the following steps:

[0047] S21, retrieve the currently queried image.

[0048] Please see details Figure 1 As shown in S11 of the illustrated embodiment, it will not be described again here.

[0049] S22, perform finger recognition on the current query image to determine the number of fingers pointing, and obtain the current query pattern corresponding to the number of fingers pointing. The number of fingers pointing refers to the number of fingers pointing at the target content in the current query image.

[0050] Specifically, S22 includes:

[0051] S221, perform finger recognition on the current query image to determine the number of fingers to be read.

[0052] Please see details Figure 1 The processing steps for recognizing the number of fingers used for reading in S12 of the illustrated embodiment will not be described in detail here.

[0053] S222, when the number of fingers pointing at the reader is detected to change from the first number to the second number, the current query mode is switched from the query mode corresponding to the first number to the query mode corresponding to the second number.

[0054] Wherein, the first quantity is the number of fingers pointing to the previous query image, and the second quantity is the number of fingers pointing to the current query image.

[0055] After the electronic device identifies the number of fingers pointing at the currently queried image as the second number, it first compares it with the first number to determine if there has been a change. If there has been no change, the query mode corresponding to the first number is directly determined as the current query mode. If there has been a change, the query mode corresponding to the second number is determined, and the current query mode is switched from the query mode corresponding to the first number to the query mode corresponding to the second number.

[0056] In some implementations, multi-finger pointing commands are defined in the electronic device. Specifically, the number of pointing fingers is positively correlated with the length of the target content in the current query mode. The target content is the query content pointed to by the pointing fingers, and the length of the target content is related to the current query mode. That is, the length of the target content is determined by the current query mode. The longer the target content length determined by the current query mode, the more pointing fingers are used; the shorter the target content length determined by the current query mode, the fewer pointing fingers are used. For example, if the current query mode is to search for a character, then a single-finger pointing is used, correspondingly defining a Chinese character or English word search; if the current query mode is to search for a word, then a two-finger pointing is used, correspondingly defining a Chinese vocabulary or English phrase search; if the current query mode is to search for a sentence, then a three-finger pointing is used, correspondingly defining a Chinese sentence or English sentence search; if the current query mode is to search for a paragraph, then a four-finger pointing is used, correspondingly defining a paragraph search.

[0057] S23, perform a query on the target content according to the current query mode and output the query results.

[0058] When a user wants to look up a word, phrase, sentence, or paragraph, they simply extend the corresponding number of fingers. The electronic device will capture and analyze the user's hand movements in real time to determine the number of fingers used for reading. Once the specific number of fingers is recognized, it automatically switches to the corresponding word search mode and performs translation, explanation, and reading aloud operations on the area pointed to by the fingers.

[0059] For further details, please see Figure 1 S13 of the illustrated embodiment will not be described again here.

[0060] The point-reading method provided in this embodiment switches the query mode corresponding to the detected change in the number of fingers used for point-reading, demonstrating real-time detection and switching during the point-reading process. Since the query mode is determined by the number of fingers used for point-reading, taking word lookup as an example, the electronic device can determine that a word lookup is in progress by using two fingers for point-reading, and then collect the word above the finger for the query. This avoids the need for quick finger swiping to determine the word to be searched when using a single finger, improving the accuracy of the query results and reducing the false recognition rate. Furthermore, setting the number of fingers used for point-reading to be positively correlated with the number of characters in the target content eliminates the need for users to memorize complex gestures, simplifying user operation and facilitating point-reading.

[0061] This embodiment provides a point-and-read method that can be used in the aforementioned electronic devices, such as mobile phones, tablets, and desk lamps. Figure 3 This is a flowchart of a point-reading method according to an embodiment of the present disclosure, such as... Figure 3As shown, the process includes the following steps:

[0062] S31, retrieve the currently queried image.

[0063] Please see details Figure 1 S11 of the illustrated embodiment will not be described again here.

[0064] S32, perform finger recognition on the current query image to determine the number of fingers pointing, so as to obtain the current query mode corresponding to the hand posture.

[0065] The "number of fingers pointing" refers to the number of fingers pointing at the target content in the currently queried image. The finger recognition result also includes the location of the pointing fingers. For further details, please refer to [link to relevant documentation]. Figure 1 S12 of the illustrated embodiment, or Figure 2 S22 of the illustrated embodiment will not be described again here.

[0066] S33, perform a query on the target content based on the current query mode and output the query results.

[0067] Specifically, S33 includes:

[0068] S331, determine the target content by using the position of the finger pointing and the current query mode.

[0069] Electronic devices, based on the current query mode, extract target content from the currently queried image using the position of the finger pointing to the text. For example, they can draw a line using the finger's position, and the range of the line is determined by the current query mode. If the current query mode is for searching a word, the line range is the area of ​​the text the finger is pointing to; if the current query mode is for searching a sentence, the line range is the area of ​​the sentence the finger is pointing to, which is determined by recognizing punctuation marks; if the current query mode is for searching a paragraph, the line range is the area of ​​the paragraph the finger is pointing to, and the paragraph range needs to be determined by recognizing paragraph markers.

[0070] S332, query the target content and output the query results.

[0071] After the electronic device identifies the target content, it queries the target content based on the offline mode or online mode described above and outputs the query results.

[0072] In some embodiments, S332 includes:

[0073] (1) Use the position of the finger to read the target content to determine whether the finger is obscuring the target content.

[0074] (2) When the target content is covered by the finger, output the query results and / or the occlusion reminder.

[0075] If a finger obscures the target content, it will affect the search results. Therefore, it is necessary to identify whether the finger is obscuring the target content before searching. As mentioned above, when identifying the target content, the electronic device draws a line above the finger. The content above the line represents the unobscured content, and the content below the line represents the obscured content. For a single character, if the proportion of the content above the line is small, it indicates that the character is obscured. Based on this, an obscuration threshold is set in the electronic device. The proportion of the content above the line is compared with the obscuration threshold. If it is less than the threshold, it indicates that the character is obscured.

[0076] When searching for sentences, the electronic device extracts the target content from the current query image based on sentence punctuation and the position of the finger tapping. If the target content is obscured, only part of the text in the sentence can be extracted. When searching for paragraphs, the electronic device first determines the scope of the paragraph based on paragraph markers and the position of the finger tapping, then locates the paragraph within the current query image and extracts text from that scope. If the text is obscured, extraction fails, and only a portion of the text can be extracted.

[0077] If the electronic device determines that the target content is obstructed, it can directly output an obstruction alert, or it can perform a query based on the recognizable content and output the query results, while also outputting an obstruction alert, and so on. The alert strategy for obstruction is set according to actual needs and is not limited here. When obstruction is detected, the query results and / or obstruction alert are output to remind the user to adjust their finger position in time.

[0078] In some alternative implementations, step (2) of S332 above includes: when the finger covering the target content, outputting the query result and / or the covering reminder based on the current query mode.

[0079] In certain query modes, occlusion does not affect search results. This is because the occluded content can be predicted based on contextual relationships, thus determining the search results. Therefore, when a finger occludes the target content, the electronic device outputs the search results and / or an occlusion warning based on the current query mode. For cases where occlusion exists but the search result can still be determined, the electronic device outputs both the search result and an occlusion warning; for cases where occlusion exists but the search result cannot be determined, the electronic device outputs an occlusion warning. Setting different occlusion warning methods for different query modes improves search efficiency and accuracy.

[0080] Specifically, when the current query mode is character or word search, an obstruction warning will be displayed via voice or pop-up. For example, the electronic device may use voice or pop-up to remind the user that "the information is obscured and cannot be recognized; please move your finger," etc. The specific content of the obstruction warning is set according to actual needs and is not limited here.

[0081] When the current query mode is sentence or paragraph search, the above-mentioned output of query results and / or occlusion warnings based on the current query mode includes:

[0082] 2.2.1) Obtain identifiable content from the target content.

[0083] 2.2.2) Match the identifiable content with the content in the preset knowledge base.

[0084] 2.2.3) Output query results and / or occlusion alerts based on the matching results.

[0085] For sentences or paragraphs, if they are partially obscured, the electronic device can identify the unobscured, identifiable content by recognizing the target content. The electronic device then matches the identifiable content with content in a preset knowledge base, calculating the similarity between the identifiable content and each item in the preset knowledge base. If the similarity is higher than a threshold, the match is successful; if the similarity is lower than the threshold, the match fails. Specifically, when the electronic device is in offline mode, the preset knowledge base is a locally maintained knowledge base; when the electronic device is in online mode, the preset knowledge base is a knowledge base maintained by a backend server.

[0086] The electronic device determines the specific content to output based on the matching result. For example, if the match is successful, the query result is output; if the match fails, an obstruction warning is output.

[0087] Since sentences or paragraphs contain a lot of text, matching is first performed based on the identifiable content in the target content, and the query results are determined through partial matching to improve query efficiency.

[0088] In step 2.2.3) above, when corresponding content is matched in the preset knowledge base, the matched content is queried, and the query result is output. Specifically, when corresponding content can be matched in the preset knowledge base using identifiable content, the query result corresponding to the matched content is output. Optionally, to improve query accuracy, the electronic device can also output an obstruction reminder.

[0089] In step 2.2.3) above, when no matching content is found in the preset knowledge base, the identifiable content is split, the splitting results are displayed, and an occlusion warning is output. In response to the selection operation of the splitting results, the content to be queried is determined; the content to be queried is queried, and the query results are output. Specifically, if the target content is occluded, preventing a matching content from being found in the preset knowledge base, the electronic device splits the identified content, for example, by word segmentation for sentences, sentence segmentation for paragraphs, etc. The splitting results are displayed on the interface of the electronic device for the user to select. The user determines the content to be queried through interactive selection with the electronic device. The electronic device queries the content to be queried and outputs the query results. Optionally, the electronic device may also output an occlusion warning.

[0090] The point-and-read method provided in this embodiment uses the position of the finger to determine the target content in the currently queried image, thereby accurately locating the target content and obtaining accurate search results. This method extends single-finger point-and-read by defining multi-finger point-and-read commands. When a user searches for a word, sentence, or paragraph, they can directly extend the corresponding number of fingers to point and read, achieving "instant search" without unnecessary selection operations, greatly improving search efficiency and user experience.

[0091] When encountering the problem of fingers obscuring target content, this embodiment provides corresponding reminder strategies based on the current query mode. Specifically, the reminder strategies for single-finger word queries and two-finger query short circuits are similar. When the target content cannot be recognized, the electronic device will remind the user via voice or pop-up window, "Information is obscured and cannot be recognized; please move your finger." The reminder strategies for three-finger sentence queries and four-finger paragraph queries are similar. When the user performs a query, the electronic device will automatically segment the target content using punctuation and information segmentation to ensure that what is recognized is a sentence or paragraph. Simultaneously, the identifiable content in the target content is matched with content in a preset knowledge base. If the similarity is higher than a threshold, the selected sentence or paragraph is determined, and preset query operations such as related explanations or readings are performed directly. If the target content is obscured by a finger, preventing a matching sentence or paragraph from being found in the preset knowledge base, the electronic device will remind the user via voice or pop-up window that "the information is obscured and cannot be recognized. Please remove your finger." The device will then perform a global sentence or paragraph split on the recognizable content, allowing the user to select the sentence or paragraph they wish to query through interactive operations with the electronic device.

[0092] This point-and-read method employs a gesture-based interaction scheme to determine the query mode, enabling convenient one-touch lookup. When users tap with their fingertips, they no longer need to frequently manually select and switch the corresponding query mode, greatly simplifying the process and improving the user experience. Furthermore, the two-finger tapping method for querying Chinese words and English phrases is more convenient and has a higher recognition rate compared to the method of quickly moving fingers to draw lines to select target content.

[0093] As a specific application example of this disclosure, the reading device is a smart desk lamp, such as... Figure 4a As shown. The smart desk lamp has a reading function button. When the user selects this button, the reading function is activated. The smart desk lamp also has an image acquisition device that can be integrated with the lamp's light tube. When the reading function is activated, the user extends a first number of fingers to point and read. The smart desk lamp captures the current query image, recognizes the number of fingers pointing, and determines the current query mode corresponding to the first number of fingers. For example, if the current query mode is "word search," the smart desk lamp outputs the query result of the word at the position of the pointing finger, via voice output. During the reading process, if the user changes from the first number of fingers to a second number, the smart desk lamp switches the current query mode from "word search" to the query mode corresponding to the second number of fingers, for example, "sentence search." The smart desk lamp analyzes the captured current query image, extracts the sentence pointed to by the finger, searches for the sentence, and outputs the query result. If the analysis of the current query image determines that there is an obstruction, the smart desk lamp combines the current query mode to output the query result and / or an obstruction reminder. For example, if the current query mode is sentence search, the smart lamp will recognize the sentence pointed to by the finger and obtain identifiable content. This identifiable content will be matched against a preset knowledge base. If a match is found, the matched content will be used to perform the query and the result will be displayed on the screen. If no match is found, an obstruction warning will be output, the identifiable content will be split and displayed, and a user-interactive selection option will be provided for the user to specify the content to be read. Figure 4b As shown, the number of fingers tapped is 3, indicating that the current query mode is sentence search. Due to the finger obstruction, only part of the content can be recognized. Therefore, the split, recognizable content is displayed on the interface for the user to select. For example... Figure 4c As shown, the splitting results are "**", "***", and "**". If the user selects one of the splitting results, Figure 4c If the user's selection is represented by a rectangle, the resulting split is used to perform a query and output the query results.

[0094] As another specific application example of this disclosure, the reading device is a smart projection device. The smart projection device is equipped with an image acquisition device. When the reading function of the smart projection device is activated, it acquires the currently queried image and performs finger recognition on the image to determine the number of fingers pointing. If the number of fingers is 1, the current query mode is determined to be a character search. The smart projection device then uses the character pointed to by the fingertip to perform the search and projects the search result onto the desktop. During the reading process, the smart projection device acquires images in real time and analyzes them. If a change in the number of fingers is detected, the current query mode is switched to the query mode corresponding to the changed number of fingers. If the switched current query mode is a sentence search, the sentence is extracted from the currently queried image, the search result is determined, and the search result is projected onto the desktop. If, during the search, the sentence cannot be recognized due to occlusion, the smart projection device segments the identifiable content of the sentence into words and projects them onto the desktop. It also extracts and analyzes the user's selection on the projected image on the desktop to determine the user's selection result. Finally, it queries the user's selection result and projects the result out.

[0095] This embodiment also provides a reading device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0096] This embodiment provides a point-and-read device, such as... Figure 5 As shown, it includes:

[0097] Module 41 is used to obtain the currently queried image;

[0098] The recognition module 42 is used to perform finger recognition on the current query image, determine the number of fingers pointing, and obtain the current query mode corresponding to the number of fingers pointing, wherein the number of fingers pointing is the number of fingers pointing to the target content in the current query image;

[0099] The query module 43 is used to query the target content according to the current query mode and output the query results.

[0100] In some implementations, the number of fingers pointing is positively correlated with the length of the target content in the current query mode.

[0101] In some implementations, the identification module 42 includes:

[0102] The first recognition unit is used to perform finger recognition on the current query image to determine the number of fingers pointing;

[0103] The switching unit is used to switch the current query mode from the query mode corresponding to the first number to the query mode corresponding to the second number when it is detected that the number of fingers pointing to the screen has changed from a first number to a second number. The first number is the number of fingers pointing to the screen corresponding to the previous query image, and the second number is the number of fingers pointing to the screen corresponding to the current query image.

[0104] In some implementations, the result of the finger recognition also includes reading the position of the finger, and the query module 43 includes:

[0105] The first determining unit is used to determine the target content by utilizing the position of the reading finger and the current query mode;

[0106] The query unit is used to query the target content and output the query results.

[0107] In some implementations, the query unit includes:

[0108] The first determining subunit is used to determine whether the reading finger is obscuring the target content by using the position relationship between the reading finger and the target content;

[0109] The output subunit is used to output the query result and / or occlusion reminder when the finger covering the target content.

[0110] In some implementations, the output subunit includes:

[0111] The first output subunit is used to output the query result and / or obstruction reminder based on the current query mode when the finger of the reader obscures the target content.

[0112] In some implementations, the first output subunit includes:

[0113] The second output subunit is used to output the obstruction reminder via voice or pop-up when the current query mode is character search or word search.

[0114] In some implementations, the first output subunit includes:

[0115] The sub-unit is used to obtain identifiable content from the target content when the current query mode is to search for a sentence or a paragraph.

[0116] A matching subunit is used to match the identifiable content with content in a preset knowledge base;

[0117] The second output subunit is used to output the query results and / or occlusion alerts based on the matching results.

[0118] In some implementations, the second output subunit includes:

[0119] The third output subunit is used to query the matched content when the corresponding content is matched in the preset knowledge base and output the query result.

[0120] In some implementations, the second output subunit includes:

[0121] The splitting subunit is used to split the identifiable content when no corresponding content is found in the preset knowledge base, display the splitting result and output the occlusion reminder;

[0122] Selecting a sub-unit is used to determine the content to be queried in response to a selection operation on the splitting result;

[0123] The fourth output subunit is used to query the content to be queried and output the query result.

[0124] In this embodiment, the reading device is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0125] Further functional descriptions of the above modules are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0126] This disclosure also provides an electronic device having the above-described features. Figure 5 The reading device shown.

[0127] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of this disclosure, such as... Figure 6As shown, the electronic device may include: at least one processor 51, such as a CPU (Central Processing Unit), at least one communication interface 53, memory 54, and at least one communication bus 52. The communication bus 52 is used to enable communication between these components. The communication interface 53 may include a display screen or a keyboard; optionally, the communication interface 53 may also include a standard wired interface or a wireless interface. The memory 54 may be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 54 may also be at least one storage device located remotely from the aforementioned processor 51. The processor 51 may be combined with... Figure 5 The described apparatus has an application program stored in memory 54, and the processor 51 calls the program code stored in memory 54 to perform any of the above method steps.

[0128] The communication bus 52 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 52 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0129] The memory 54 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 54 may also include a combination of the above types of memory.

[0130] The processor 51 can be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP.

[0131] The processor 51 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0132] Optionally, the memory 54 is also used to store program instructions. The processor 51 can invoke the program instructions to implement the point-reading method as shown in any embodiment of this application.

[0133] This disclosure also provides a non-transitory computer storage medium storing computer-executable instructions that can execute the point-reading method in any of the above method embodiments. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0134] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0135] It is understood that in the specific embodiments of this disclosure, data related to user information and interactive content are involved. When the above embodiments of this disclosure are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0136] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A point-and-read method, characterized in that, include: Get the currently queried image; Finger recognition is performed on the current query image to determine the number of fingers pointing, so as to obtain the current query mode corresponding to the number of fingers pointing. The number of fingers pointing refers to the number of fingers pointing to the target content in the current query image. The number of fingers pointing is positively correlated with the length of the target content in the current query mode. The target content is queried according to the current query mode, and the query results are output.

2. The method according to claim 1, characterized in that, The step of performing finger recognition on the current query image to determine the number of fingers pointing, and obtaining the current query pattern corresponding to the number of fingers pointing, includes: The number of fingers used for pointing is determined by finger recognition in the current query image; When the number of fingers pointing at the screen is detected to change from the first number to the second number, the current query mode is switched from the query mode corresponding to the first number to the query mode corresponding to the second number. The first number is the number of fingers pointing at the screen corresponding to the previous query image, and the second number is the number of fingers pointing at the screen corresponding to the current query image.

3. The method according to claim 1, characterized in that, The result of the finger recognition also includes reading the position of the finger, and the querying of the target content according to the current query mode and outputting the query result includes: The target content is determined using the position of the finger being tapped and the current query mode. Perform a query on the target content and output the query results.

4. The method according to claim 3, characterized in that, The querying of the target content and the output of the query results include: By using the position of the reading finger and the position of the target content, it can be determined whether the reading finger is obscuring the target content; When the finger used for pointing obscures the target content, the query result and / or obscuration reminder are output.

5. The method according to claim 4, characterized in that, When the finger used for tapping obscures the target content, the query result and / or obscuration warning are output, including: When the finger used for pointing obscures the target content, the query result and / or obscuration reminder are output based on the current query mode.

6. The method according to claim 5, characterized in that, The step of outputting the query results and / or occlusion alerts based on the current query mode includes: When the current query mode is character or word search, the obstruction reminder is output via voice or pop-up window.

7. The method according to claim 5, characterized in that, The step of outputting the query results and / or occlusion alerts based on the current query mode includes: When the current query mode is to search for a sentence or a paragraph, obtain the identifiable content in the target content; The identifiable content is matched with the content in a preset knowledge base; Output the query results and / or occlusion alerts based on the matching results.

8. The method according to claim 7, characterized in that, The step of outputting the query results and / or occlusion alerts based on the matching results includes: When a corresponding content is found in the preset knowledge base, the matched content is queried, and the query result is output.

9. The method according to claim 7, characterized in that, The step of outputting the query results and / or occlusion alerts based on the matching results includes: When no matching content is found in the preset knowledge base, the identifiable content is split, the splitting result is displayed, and the occlusion reminder is output. In response to the selection operation of the splitting results, the content to be queried is determined; Perform a query on the content to be queried and output the query results.

10. A reading device, characterized in that, include: The acquisition module is used to acquire the currently queried image; The recognition module is used to perform finger recognition on the current query image, determine the number of fingers pointing, and obtain the current query mode corresponding to the number of fingers pointing. The number of fingers pointing refers to the number of fingers pointing to the target content in the current query image, and the number of fingers pointing is positively correlated with the length of the target content in the current query mode. The query module is used to query the target content according to the current query mode and output the query results.

11. An electronic device, characterized in that, include: A memory and a processor are interconnected, the memory stores computer instructions, and the processor executes the point-reading method according to any one of claims 1-9 by executing the computer instructions.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the point-reading method according to any one of claims 1-9.

Images