Picture recognition method and system for a scanning pen

Abstract

The embodiment of the present application relates to the technical field of character recognition, and specifically discloses a picture recognition method and system for a scanning pen, comprising: determining a character picture containing one character and a sentence picture containing multiple characters according to a picture sequence output by a camera, performing character recognition on each character picture to obtain N candidate characters of each character picture and the probability of each candidate character, inputting the sentence picture into a pre-trained sentence recognition model to obtain a candidate sentence, inputting the candidate sentence into a semantic analysis model to obtain a first score of the candidate sentence, calculating a second score of the candidate sentence using the probability of each character in the candidate sentence, calculating the confidence of the candidate sentence according to the first score and the second score, and determining the candidate sentence as a target output sentence when the confidence is greater than a preset threshold, the embodiment is fast and efficient in recognizing character pictures and sentence pictures, and the accuracy is high in determining the target output sentence through the confidence of the candidate sentence.

Description

Technical Field

[0001] The present invention belongs to the technical field of text recognition, and in particular relates to an image recognition method and system for a scanning pen. Background Art

[0002] Scanning pens are widely used in reading, learning and other aspects of life because they can recognize printed text and display it, voice broadcast it, and translate it by sliding it over it.

[0003] Currently, after scanning is completed, for example, after the pen is lifted, the video data recorded by the camera on the scanning pen is input into the text recognition model to identify the scanned sentence. Since the recognition is based on video data, on the one hand, the sentence recognition speed is slow and inefficient, and on the other hand, the accuracy of the output sentence cannot be guaranteed. Summary of the Invention

[0004] The embodiment of the present invention aims to provide an image recognition method and system for a scanning pen, aiming to solve the problems of slow text recognition speed, low efficiency and inability to guarantee accuracy of scanning pens.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A method for image recognition using a scanning pen, applied to a scanning pen equipped with a camera, specifically comprising the following steps:

[0007] Obtaining a sequence of images output by the camera during scanning by the scanning pen, and determining a plurality of text images and a sentence image based on the sequence of images, wherein the text image is an image including a single character, and the sentence image is an image of a sentence composed of multiple characters;

[0008] Perform text recognition on each text image to obtain N candidate texts for each text image, as well as the probability of each candidate text;

[0009] Inputting the sentence image into a pre-trained sentence recognition model to obtain candidate sentences;

[0010] Inputting the candidate sentence into a semantic analysis model to obtain a first score for the candidate sentence;

[0011] Calculating a second score for the candidate sentence using the probability of each word in the candidate sentence;

[0012] Calculating the confidence of the candidate sentence according to the first score and the second score;

[0013] When the confidence of the candidate sentence is greater than a preset threshold, the candidate sentence is determined as the target output sentence.

[0014] As a further limitation of the embodiment of the present invention, determining a plurality of text images and a sentence image according to the image sequence specifically includes the following steps:

[0015] When the camera outputs a picture sequence, the picture sequence is input into a text picture generation model to obtain a plurality of text pictures;

[0016] Multiple text images are spliced ​​together to obtain sentence images.

[0017] As a further limitation of the embodiment of the present invention, performing text recognition on each text image to obtain N candidate texts for the text image and the probability of each candidate text specifically includes the following steps:

[0018] Each text image is input into a text recognition model to obtain N candidate texts for the text image and the probability of each candidate text.

[0019] As a further limitation of the embodiment of the present invention, calculating the second score of the candidate sentence using the probability of each character in the candidate sentence specifically includes the following steps:

[0020] Determine a target text from the text of the candidate sentence, wherein the target text is a candidate text of the text image;

[0021] Calculate the average of the probabilities of multiple target words;

[0022] Determining the number of the target characters in the candidate sentences and the total number of the candidate sentences, and calculating the ratio of the number of the target characters to the total number to obtain a candidate character ratio;

[0023] A product of the average value and the ratio is calculated as a second score of the candidate sentence.

[0024] As a further limitation of this embodiment of the present invention, calculating the confidence of the candidate sentence based on the first score and the second score includes:

[0025] Calculate the difference between 1 and the candidate character ratio, and calculate a first product of the difference and the first score;

[0026] Calculate a second product of the candidate character ratio and the second score;

[0027] The sum of the first product and the second product is calculated to obtain the confidence of the candidate sentence.

[0028] As a further limitation of the embodiment of the present invention, the following steps are specifically included:

[0029] When the confidence of the candidate sentence is less than or equal to a preset threshold, the candidate sentence is updated using the candidate text, and the process returns to the step of inputting the candidate sentence into the semantic analysis model to obtain the first score of the candidate sentence.

[0030] As a further limitation of the embodiment of the present invention, when the confidence of the candidate sentence is less than or equal to a preset threshold, updating the candidate sentence with the candidate text specifically includes the following steps:

[0031] Determining a target text image for each character in the candidate sentence, and determining whether the character is the candidate character with the highest probability among N candidate characters in the target text image;

[0032] If not, the candidate text with the highest probability is used to replace the text in the candidate sentence to update the candidate sentence.

[0033] As a further limitation of the embodiment of the present invention, after replacing the characters in the candidate sentence with the candidate characters with the highest probability to update the candidate sentence, the method further specifically includes the following steps:

[0034] When the confidence levels of multiple candidate sentences are all less than a preset threshold, the m candidate sentences ranked highest in confidence level are displayed on the display screen;

[0035] In response to a user's selection operation on a candidate sentence, the candidate sentence selected by the selection operation is determined as a target output sentence.

[0036] As a further limitation of an embodiment of the present invention, the scanning pen further includes a voice broadcasting device, which, after determining the candidate sentence as a target output sentence, further specifically includes the following steps:

[0037] When a voice broadcast instruction is received, the target output sentence is broadcasted in a preset language.

[0038] An image recognition system for a scanning pen, applied to a scanning pen equipped with a camera, specifically includes the following units:

[0039] An image acquisition unit is configured to acquire an image sequence output by the camera during scanning by the scanning pen, and determine a plurality of text images and a sentence image based on the image sequence, wherein the text image is an image including a single character, and the sentence image is an image of a sentence composed of multiple characters;

[0040] A text recognition unit is used to perform text recognition on each text image to obtain N candidate characters for each text image, as well as the probability of each candidate character;

[0041] A candidate sentence recognition unit, configured to input the sentence image into a pre-trained sentence recognition model to obtain candidate sentences;

[0042] a first score determination unit, configured to input the candidate sentence into a semantic analysis model to obtain a first score for the candidate sentence;

[0043] a second score determination unit, configured to calculate a second score of the candidate sentence using the probability of each word in the candidate sentence;

[0044] a confidence determination unit, configured to calculate the confidence of the candidate sentence based on the first score and the second score;

[0045] The target output sentence determining unit is configured to determine the candidate sentence as the target output sentence when the confidence of the candidate sentence is greater than a preset threshold.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The scanning pen of an embodiment of the present invention determines, based on a sequence of images output by a camera, a text image containing a single character and a sentence image containing multiple characters of the scanned sentence. It then performs text recognition on each text image to obtain N candidate characters and the probability of each candidate character. The sentence image is then input into a pre-trained sentence recognition model to obtain a candidate sentence. The candidate sentence is then input into a semantic analysis model to obtain a first score for the candidate sentence. The probability of each character in the candidate sentence is used to calculate a second score for the candidate sentence. The confidence of the candidate sentence is calculated based on the first and second scores. When the confidence is greater than a preset threshold, the candidate sentence is determined as the target output sentence. This improves the accuracy of the sentences output by the scanning pen by rapidly and efficiently recognizing text images and sentence images. Furthermore, the second score is calculated based on the probabilities of the characters in the candidate sentence, the first score of the candidate sentence is predicted using a semantic analysis model, and the confidence of the candidate sentence is calculated based on the first and second scores. When the confidence is greater than the threshold, the candidate sentence is determined as the target output sentence. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention.

[0049] Figure 1 A flowchart of an image recognition method for a scanning pen provided by an embodiment of the present invention is shown.

[0050] Figure 2 A schematic diagram showing candidate sentences, candidate text images, and sentence images;

[0051] Figure 3 The figure shows an application architecture diagram of an image recognition system for a scanning pen provided in an embodiment of the present invention. Detailed implementation manners

[0052] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0053] Figure 1 The figure shows a flowchart of an image recognition method for a scanning pen provided in an embodiment of the present invention. Specifically, the image recognition method for a scanning pen in an embodiment of the present invention is applied to a scanning pen provided with a camera, and specifically includes the following steps:

[0054] Step S101, obtain a sequence of pictures output by the camera during the scanning process of the scanning pen, and determine multiple text pictures and one sentence picture according to the picture sequence. A text picture is a picture including one character, and a sentence picture is a picture of a sentence composed of multiple characters.

[0055] In this embodiment, a text picture may be a picture including one character. The character should be the character defined in a broad sense, that is, not limited by the language. In one example, when the character is a Chinese character, the text picture may be a picture including one Chinese character or punctuation mark or other symbol. When the character is an English character, the text picture may be a picture including one English word or punctuation mark or other symbol. A sentence picture is a picture including all the characters of the sentence scanned by the scanning pen. Taking the sentence "Tomorrow is the weekend" scanned by the scanning pen on a piece of paper as an example, the text pictures may be pictures of the characters "明", "天", "就", "是", "周", "末", "了", and the sentence picture may be a picture including the sentence "Tomorrow is the weekend".

[0056] In one embodiment, when the camera outputs a picture sequence, the picture sequence may be input into a text picture generation model to obtain multiple text pictures, and the multiple text pictures are spliced to obtain a sentence picture. Exemplarily, when training the text picture generation model, training data may be first obtained. The training data may be a picture sequence obtained by scanning a sentence. In this picture sequence, the starting picture and ending picture of each character and the text picture of the character are marked. The training data is input into the text picture generation model to obtain predicted text pictures, and then the loss rate is calculated to update the model, so that the model learns the ability to generate text pictures from a picture sequence.

[0057] After inputting a sequence of pictures into a text-picture generation model, multiple text pictures can be obtained, and they can be spliced in the order of generation of the text pictures, that is, a sentence picture is obtained.

[0058] In another alternative embodiment, when the camera outputs each picture, the picture can be binarized so that the pixels containing text handwriting in the picture are filled with black, and the pixel columns without black pixels in each picture are counted. When the number of pixel columns without black pixels is greater than the threshold, it indicates that the interval between the collected texts is detected. The picture sequence between the two intervals is determined as the picture sequence of one text. The picture sequence of this text is used to splice out a text picture, and the text pictures are spliced into a sentence picture.

[0059] As Figure 2 shown, for the picture sequence output by scanning the sentence "Tomorrow is the weekend" with a scanning pen, text pictures 1-7 are determined, which respectively correspond to the pictures of each text in the scanned sentence "Tomorrow is the weekend". Splicing text pictures 1-7 can obtain a sentence picture.

[0060] Step S102: Perform text recognition on each text picture to obtain N candidate texts for each text picture and the probability of each candidate text.

[0061] In one embodiment, a text recognition model can be pre-trained. This text recognition model can recognize the text in the input picture. Each text picture is input into the text recognition model to obtain N candidate texts for the text picture and the probability of each candidate text. The training method of the text recognition model can refer to the prior art and will not be elaborated here.

[0062] In another embodiment, the similarity between each text picture and the reference pictures in a preset picture library can be calculated, and the texts associated with the N reference pictures with the highest similarity are used as candidate texts, and the similarity is used as the probability of the candidate texts.

[0063] As Figure 2 shown, the candidate texts for the text picture of the character "明" include "明", "朋", and "胡", and the probabilities are 0.7, 0.2, and 0.1 respectively. The candidate texts for the text picture of the character "天" include "天", "夫", and "夭", and the probabilities are 0.6, 0.3, and 0.1 respectively. The candidate texts and probabilities of other text pictures are as Figure 2 shown.

[0064] Step S103: Input the sentence picture into a pre-trained sentence recognition model to obtain candidate sentences.

[0065] In this embodiment, a sentence recognition model can be pre-trained. The training data for the sentence recognition model is pictures of sentences in various sentence patterns, and the sentences, sentence patterns are labeled for the pictures, such as labeling the subject, predicate, object, adverb, adjective, etc., so that the sentence recognition model can learn the literal shape and sentence pattern of the sentence through the picture at the same time, making the recognition of candidate sentences more accurate. The training method of the sentence recognition model can refer to the existing model training methods. For example, after inputting the training sentence into the model, the predicted sentence and the sentence pattern are obtained. The loss rate is calculated through the predicted sentence and sentence pattern and the labeled sentence and sentence pattern to adjust the model parameters. The loss function can be a mean square error loss function, a cross-entropy loss function, etc. The method of adjusting the model parameters can adopt various gradient descent methods, which will not be elaborated here.

[0066] As Figure 2 shown, after inputting the sentence picture spliced by text pictures 1-7 into the sentence recognition model, the output candidate sentence is "Mingfu jiu zu zhou wei ding".

[0067] Step S104, input the candidate sentence into the semantic analysis model to obtain the first score of the candidate sentence.

[0068] The semantic analysis model can be a model that analyzes the semantic score of the input sentence. The semantic score is used to measure the ability of the input sentence to accurately express semantics. When training the semantic analysis model, positive samples and negative samples can be prepared first. Exemplarily, for a training sentence, the original training sentence is used as the positive sample, and the词性 of each word in the training sentence and the sentence pattern are labeled, and a score (high score, such as 1 point) is assigned. One or more words in the positive sample sentence are replaced, such as replacing them with words of the same词性 or words of different词性, to obtain a negative sample, and the词性, sentence pattern and score (low score, such as 0 point) are labeled. The negative sample is a sentence that cannot express a definite semantics, and the positive sample is a sentence that can express the correct semantics. The semantic analysis model is trained through the positive sample and negative sample, so that the semantic analysis model learns the ability to analyze whether the sentence accurately expresses semantics according to the词性 of the words and the sentence structure in the sentence. In this embodiment, the candidate sentence is input into the semantic analysis model to obtain the first score of the candidate sentence. The first score represents the ability of the candidate sentence to accurately express semantics.

[0069] In another embodiment, the sentence pattern structure of the candidate sentence and the词性 of the words and words in the candidate sentence can also be extracted to score the candidate sentence. Among them, scoring rules for various sentence patterns can be preset in advance, and the candidate sentence is scored according to the scoring rules to obtain the first score.

[0070] Step S105, calculate the second score of the candidate sentence by using the probabilities of each character in the candidate sentence.

[0071] In one embodiment, the target text can be determined from the text of the candidate statement, where the target text is the text in the candidate statement and is the candidate text of the text picture, and the average value of the probabilities of multiple target texts is calculated, the number of target texts in the candidate statement and the total number of the candidate statement are determined, and the ratio of the number of target texts to the total number is calculated to obtain the proportion of candidate texts, and the product of the average value and the ratio is calculated as the second score of the candidate statement.

[0072] Exemplarily, as Figure 2 shown, in the candidate statement "Mingfujiuzhouweiding", except for the character "zu", other characters are the characters in the candidate texts of text pictures 1-7 and have corresponding probabilities. The average value of the probabilities of "Ming", "fu", "jiu", "zhou", "wei", and "ding" can be calculated. In Figure 2 the example, the number of target texts with probabilities is 6, the total number of characters in the candidate statement "Mingfujiuzhouweiding" is 7, the ratio is 6 / 7, and the product of the average value of the probabilities and the ratio is calculated as the second score of the candidate statement. This second score represents the accuracy of the character shape of the candidate statement.

[0073] In another embodiment, the sum value of the probabilities of the target texts can also be directly calculated, and the product of the sum value and the ratio is calculated as the second score of the candidate statement.

[0074] Step S106, calculate the confidence of the candidate statement according to the first score and the second score.

[0075] In one embodiment, the difference between 1 and the proportion of candidate texts can be calculated, and the first product of the difference and the first score is calculated, the second product of the proportion of candidate texts and the second score is calculated, and the sum value of the first product and the second product is calculated to obtain the confidence of the candidate statement.

[0076] From the above calculation method of the confidence, the larger the proportion of candidate texts, the larger the second score, which means that the larger the proportion of candidate texts that are simultaneously candidate texts of text pictures in the candidate statement recognized by the statement recognition model, and the higher the possibility of the candidate text with the maximum probability of being a text picture. That is, the confidence of the candidate statement calculated after double recognition by the text recognition model and the statement recognition model is higher, and the candidate statement is more accurate.

[0077] In another embodiment, the sum value of the first score and the second score can also be calculated as the confidence of the candidate statement, or the product of the first score and the second score can be calculated as the confidence of the candidate statement. This embodiment does not limit the method of calculating the confidence of the candidate statement through the first score and the second score.

[0078] Step S107, when the confidence of the candidate statement is greater than the preset threshold, determine the candidate statement as the target output statement.

[0079] When the confidence of the candidate statement is greater than the preset threshold, it is determined that the candidate statement is the statement scanned by the scanning pen. When a voice broadcast instruction is received, the target output statement is broadcast in the voice of the preset language.

[0080] In another embodiment, when the confidence of the candidate statement is less than or equal to the preset threshold, the candidate statement is updated with the candidate text. Exemplarily, the target text picture of each character in the candidate statement can be determined, and it can be determined whether the character is the candidate character with the highest probability among N candidate characters of the target text picture. If not, the candidate character with the highest probability is used to replace the character in the candidate statement to update the candidate statement. After obtaining the updated candidate statement, return to step S104 until the confidence is greater than the threshold.

[0081] Exemplarily, as Figure 2 shown, when updating the candidate statement, the character "夫" in the candidate statement is not the candidate character with the highest probability among the candidate characters of text picture 2, and the candidate character with the highest probability is the character "天". Similarly, the characters "足", "未", and "丁" in the candidate statement are not the candidate characters with the highest probability of the corresponding text pictures. The candidate character with the highest probability can be used to replace the characters in the candidate statement to obtain the updated candidate statement "明天就是周末了". The updated candidate statement "明天就是周末了" is input into the semantic analysis model to obtain the first score, and the second score is calculated using the probabilities of each character. It can be seen that the updated candidate statement can express the semantics more accurately, its first score is higher, and at the same time the second score is also higher, and the confidence is also higher, thereby improving the accuracy of the scanning pen for scanning statements.

[0082] In another embodiment, when the confidence of the updated multiple candidate statements is less than the preset threshold, the m candidate statements with the highest confidence ranking are displayed on the display screen. In response to the user's selection operation on the candidate statement, the candidate statement selected by the selection operation is determined as the target output statement. This avoids providing m candidate statements for the user to select when the scanned statement is not clear enough or the scanned statement is in a handwritten font that is not standard and difficult to recognize.

[0083] The scanning pen of an embodiment of the present invention determines, based on a sequence of images output by a camera, a text image containing a single character and a sentence image containing multiple characters of the scanned sentence. It then performs text recognition on each text image to obtain N candidate characters and the probability of each candidate character. The sentence image is then input into a pre-trained sentence recognition model to obtain a candidate sentence. The candidate sentence is then input into a semantic analysis model to obtain a first score for the candidate sentence. The probability of each character in the candidate sentence is used to calculate a second score for the candidate sentence. The confidence of the candidate sentence is calculated based on the first and second scores. When the confidence is greater than a preset threshold, the candidate sentence is determined as the target output sentence. This improves the accuracy of the sentences output by the scanning pen by rapidly and efficiently recognizing text images and sentence images. Furthermore, the second score is calculated based on the probabilities of the characters in the candidate sentence, the first score of the candidate sentence is predicted using a semantic analysis model, and the confidence of the candidate sentence is calculated based on the first and second scores. When the confidence is greater than the threshold, the candidate sentence is determined as the target output sentence.

[0084] Figure 3 The following diagram shows the application architecture of an image recognition system for a scanning pen provided by an embodiment of the present invention. The image recognition system for a scanning pen of this embodiment is applied to a scanning pen equipped with a camera and specifically includes the following units:

[0085] The image acquisition unit 301 is configured to acquire an image sequence output by the camera during scanning by the scanning pen, and determine a plurality of text images and a sentence image based on the image sequence, wherein the text image is an image including a single character, and the sentence image is an image of a sentence composed of multiple characters;

[0086] A character recognition unit 302 is configured to perform character recognition on each character image to obtain N candidate characters for each character image and the probability of each candidate character;

[0087] A candidate sentence recognition unit 303 is configured to input the sentence image into a pre-trained sentence recognition model to obtain a candidate sentence;

[0088] A first score determination unit 304 is configured to input the candidate sentence into a semantic analysis model to obtain a first score for the candidate sentence;

[0089] A second score determination unit 305 is configured to calculate a second score of the candidate sentence using the probability of each character in the candidate sentence;

[0090] a confidence determination unit 306, configured to calculate the confidence of the candidate sentence based on the first score and the second score;

[0091] The target output sentence determining unit 307 is configured to determine the candidate sentence as the target output sentence when the confidence level of the candidate sentence is greater than a preset threshold.

[0092] As a further limitation of the embodiment of the present invention, the image acquisition unit 301 specifically includes the following modules:

[0093] A text image acquisition module, configured to input a sequence of images into a text image generation model when the camera outputs the sequence of images, to obtain a plurality of text images;

[0094] The sentence image acquisition module is used to splice multiple text images to obtain sentence images.

[0095] As a further limitation of this embodiment of the present invention, the text recognition unit 302 specifically includes the following modules:

[0096] The text recognition model recognition module is used to input each text image into the text recognition model to obtain N candidate characters for the text image and the probability of each candidate character.

[0097] As a further limitation of this embodiment of the present invention, the second score determination unit 305 specifically includes the following modules:

[0098] A target text determination module, configured to determine a target text from the text in the candidate sentences, wherein the target text is a candidate text in the text image;

[0099] A probability average calculation module is used to calculate the average value of the probabilities of multiple target characters;

[0100] a character ratio calculation module, configured to determine the number of the target characters in the candidate sentence and the total number of the candidate sentences, and calculate the ratio of the number of the target characters to the total number to obtain the candidate character ratio;

[0101] A second score calculation module is configured to calculate a product of the average value and the ratio as a second score of the candidate sentence.

[0102] As a further limitation of this embodiment of the present invention, the confidence determination unit 306 includes:

[0103] a difference and first product calculation module, configured to calculate a difference between 1 and the candidate character proportion, and calculate a first product of the difference and the first score;

[0104] A second product calculation module, configured to calculate a second product of the candidate character ratio and the second score;

[0105] The confidence calculation module is configured to calculate the sum of the first product and the second product to obtain the confidence of the candidate sentence.

[0106] As a further limitation of the embodiment of the present invention, the following units are specifically included:

[0107] The candidate sentence updating module is configured to update the candidate sentence with the candidate text when the confidence of the candidate sentence is less than or equal to a preset threshold, and return the result to the first score determining unit 304 .

[0108] As a further limitation of the embodiment of the present invention, the candidate statement updating module specifically includes the following submodules:

[0109] A candidate character judgment submodule is used to determine a target character image for each character in the candidate sentence and determine whether the character is the candidate character with the highest probability among the N candidate characters in the target character image;

[0110] The candidate character replacement submodule is used to replace the characters in the candidate sentence with the candidate character with the highest probability to update the candidate sentence.

[0111] As a further limitation of the embodiment of the present invention, the following units are further specifically included:

[0112] A candidate sentence display unit, configured to display m candidate sentences ranked highest in confidence on a display screen when the confidence levels of multiple candidate sentences are all lower than a preset threshold;

[0113] The target output sentence determining unit is configured to respond to a user's selection operation on a candidate sentence and determine the candidate sentence selected by the selection operation as the target output sentence.

[0114] As a further limitation of the embodiment of the present invention, the following units are further specifically included:

[0115] The target output sentence broadcasting unit is used to broadcast the target output sentence in a preset language when receiving a voice broadcast instruction.

[0116] It should be understood that, although the various steps in the flow chart of each embodiment of the present invention are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0117] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When executed, the program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0118] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The above embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0120] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for image recognition using a scanning pen, characterized in that: Applicable to a scanning pen equipped with a camera, specifically including the following steps: Obtaining a sequence of images output by the camera during scanning by the scanning pen, and determining a plurality of text images and a sentence image based on the sequence of images, wherein the text image is an image including a single character, and the sentence image is an image of a sentence composed of multiple characters; Perform text recognition on each text image to obtain N candidate texts for each text image, as well as the probability of each candidate text; Inputting the sentence image into a pre-trained sentence recognition model to obtain candidate sentences; Inputting the candidate sentence into a semantic analysis model to obtain a first score for the candidate sentence; Calculating a second score for the candidate sentence using the probability of each word in the candidate sentence; Calculating the confidence of the candidate sentence according to the first score and the second score; When the confidence of the candidate sentence is greater than a preset threshold, the candidate sentence is determined as the target output sentence.

2. The image recognition method for a scanning pen according to claim 1, characterized in that: Determining a plurality of text images and a sentence image according to the image sequence specifically includes the following steps: When the camera outputs a picture sequence, the picture sequence is input into a text picture generation model to obtain a plurality of text pictures; Multiple text images are spliced ​​together to obtain sentence images.

3. The image recognition method for a scanning pen according to claim 1, characterized in that: Performing text recognition on each text image to obtain N candidate texts for the text image and the probability of each candidate text specifically includes the following steps: Each text image is input into a text recognition model to obtain N candidate texts for the text image and the probability of each candidate text.

4. The image recognition method for a scanning pen according to claim 1, characterized in that: Calculating a second score of the candidate sentence using the probability of each word in the candidate sentence specifically includes the following steps: Determine a target text from the text of the candidate sentence, wherein the target text is a candidate text of the text image; Calculate the average of the probabilities of multiple target words; Determining the number of the target characters in the candidate sentences and the total number of the candidate sentences, and calculating the ratio of the number of the target characters to the total number to obtain a candidate character ratio; A product of the average value and the ratio is calculated as a second score of the candidate sentence.

5. The image recognition method for a scanning pen according to claim 4, characterized in that: Calculating the confidence of the candidate sentence according to the first score and the second score includes: Calculate the difference between 1 and the candidate character ratio, and calculate a first product of the difference and the first score; Calculate a second product of the candidate character ratio and the second score; The sum of the first product and the second product is calculated to obtain the confidence of the candidate sentence.

6. The image recognition method for a scanning pen according to any one of claims 1 to 5, characterized in that: The specific steps include: When the confidence of the candidate sentence is less than or equal to a preset threshold, the candidate sentence is updated using the candidate text, and the process returns to the step of inputting the candidate sentence into the semantic analysis model to obtain the first score of the candidate sentence.

7. The image recognition method for a scanning pen according to claim 6, characterized in that: When the confidence of the candidate sentence is less than or equal to a preset threshold, updating the candidate sentence with the candidate text specifically includes the following steps: Determining a target text image for each character in the candidate sentence, and determining whether the character is the candidate character with the highest probability among N candidate characters in the target text image; If not, the candidate text with the highest probability is used to replace the text in the candidate sentence to update the candidate sentence.

8. The image recognition method for a scanning pen according to claim 6, characterized in that: After replacing the words in the candidate sentence with the candidate word with the highest probability to update the candidate sentence, the method further specifically includes the following steps: When the confidence levels of multiple candidate sentences are all less than a preset threshold, the m candidate sentences ranked highest in confidence level are displayed on the display screen; In response to a user's selection operation on a candidate sentence, the candidate sentence selected by the selection operation is determined as a target output sentence.

9. The image recognition method for a scanning pen according to any one of claims 1 to 5, characterized in that: The scanning pen further includes a voice broadcasting device, which, after determining the candidate sentence as the target output sentence, further specifically includes the following steps: When a voice broadcast instruction is received, the target output sentence is broadcasted in a preset language.

10. An image recognition system for a scanning pen, characterized in that: Applicable to a scanning pen equipped with a camera, specifically including the following units: An image acquisition unit is configured to acquire an image sequence output by the camera during scanning by the scanning pen, and determine a plurality of text images and a sentence image based on the image sequence, wherein the text image is an image including a single character, and the sentence image is an image of a sentence composed of multiple characters; A text recognition unit is used to perform text recognition on each text image to obtain N candidate characters for each text image, as well as the probability of each candidate character; A candidate sentence recognition unit, configured to input the sentence image into a pre-trained sentence recognition model to obtain candidate sentences; a first score determination unit, configured to input the candidate sentence into a semantic analysis model to obtain a first score for the candidate sentence; a second score determination unit, configured to calculate a second score of the candidate sentence using the probability of each word in the candidate sentence; a confidence determination unit, configured to calculate the confidence of the candidate sentence based on the first score and the second score; The target output sentence determining unit is configured to determine the candidate sentence as the target output sentence when the confidence of the candidate sentence is greater than a preset threshold.

Images