Method and device for improving reliability of picture book point reading recognition results

Abstract

The present invention relates to a method and device for improving the reliability of picture book point reading recognition results, so that in the picture book point reading recognition process, by using multiple consecutive (2 or more than 2) recognition results, the recognition results are integrated and extremely high-precision results are output to the user, thereby improving the user experience.

Description

Technical Field

[0001] The present invention relates to a method and a device for improving the reliability of picture book point reading recognition results, which can effectively improve the accuracy of identifying picture books and other books by using devices such as smart terminals. Background Art

[0002] For books such as picture books (each page has a large picture with text), use a smartphone or other device with a camera and image recognition software to automatically locate the page number of a page in the book and the position of the finger, and automatically play the multimedia resources at the position where the finger is pointing.

[0003] The whole device consists of four parts: a terminal such as a smart phone, software installed on the terminal, a reflector connected to the front camera of the terminal, and a bracket that fixes the terminal at a specific angle. The pages of picture books and other books placed on the table in front of the bracket are photographed by the front camera of the terminal through the reflector. The image recognition software automatically locates the page number of a certain page in the book and the position where the finger is pointing, and automatically plays the multimedia resources at the position where the finger is pointing.

[0004] The image recognition software contained in the picture book reading device has the main function of using the book page image (which may include fingers) taken by the camera of the terminal device, using the relevant image recognition models and algorithms of artificial intelligence to identify the book page number corresponding to the book page image, and detect the corresponding position of the user's finger in the book page, so as to obtain the multimedia resources corresponding to the book page or finger position, and automatically play them to the user.

[0005] Because ① there may be similar pages in a book, ② there may be geometric deformations such as warping when the book pages are placed, ③ there are many uncontrollable factors such as lens deformation and ambient light influence when the camera captures the book page image, and ④ the various current artificial intelligence recognition algorithms are not perfect and still have a certain error rate; these factors work together, resulting in the current recognition algorithm's recognition accuracy for book pages and the detection accuracy of the presence and position of fingers failing to achieve ideal performance. A single recognition can generally only achieve an accuracy of about 90%. If a single recognition result is used directly, the probability of recognition error when the user uses it will reach about 10%, and the user experience will be very poor.

[0006] Therefore, how to prevent recognition errors from affecting user experience and improve user experience is a technical problem that urgently needs to be solved. Summary of the invention

[0007] The purpose of the present invention is to design a method and device for improving the reliability of picture book point reading recognition results, so that recognition errors in the picture book point reading recognition process do not affect the user experience, thereby improving the user experience.

[0008] The method for improving the reliability of picture book point reading recognition results in the terminal includes the steps of: 1) turning on the book point reading function and setting the initial value of the page state cp1 to be empty;

[0009] 2) Counter initialization, set counter n = 1;

[0010] 3) Setting a timer interval ⊿t, and taking a book page image through a camera in the terminal at a time interval ⊿t or after the last recognition is completed, and performing book page recognition on the captured image, and detecting the presence and position of a finger in the captured image; wherein the book page corresponding to the recognition is recorded as p1;

[0011] 4) Compare the book page p1 with the page state cp1; determine whether p1 is equal to cp1; if the determination result is no, execute step 5); if the determination result is yes, execute step 6);

[0012] 5) Set the cp1 to p1 and return to step 2);

[0013] 6) adding 1 to the n;

[0014] 7) Determine whether n is greater than or equal to a preset number of times N, where N is greater than or equal to 2; if the determination result is no, return to step 3); if the determination result is yes, execute step 8);

[0015] 8) Output the reliable book page recognition result as p1.

[0016] The method further includes: after obtaining a reliable book page recognition result output, updating the current book page status recorded in the system.

[0017] Furthermore, if the recognition result of the current page is different from the previous N-1 consecutive page recognition results, the current recognition result is not output or is marked as unreliable.

[0018] The step 3) detects the presence and position of a finger in the captured image, specifically including: if a finger is detected in the current image, and the finger is detected in the previous N-1 consecutive images, and the finger position (x, y) detected this time is the same as the finger position (x, y) detected in the previous N-1 consecutive images. i ,y i ) i＝1,…,N-1 The deviation between i =||(x, y)-(x i ,y i)|| are both less than a pre-specified fixed value DP, the finger detection result is given as: the finger is reliably detected and the finger position is (x, y); otherwise, the finger detection result is marked as no finger is detected or the detection result is unreliable.

[0019] Furthermore, the playing status of the multimedia at different positions of the corresponding book page is determined according to the page status and the change in the finger status and position.

[0020] In addition, a device for improving the reliability of picture book point reading recognition results in a terminal is also provided, the device comprising a processor and a memory, the memory storing program codes, and when the program codes are executed by the processor, they are used to implement the following functions:

[0021] 1) Enable the book reading function and set the initial value of page status cp1 to empty;

[0022] 2) Counter initialization, set counter n = 1;

[0023] 3) Setting a timer interval ⊿t, and taking a book page image through a camera in the terminal at a time interval ⊿t or after the last recognition is completed, and performing book page recognition on the captured image, and detecting the presence and position of a finger in the captured image; wherein the book page corresponding to the recognition is recorded as p1;

[0024] 4) Compare the book page p1 with the page state cp1; determine whether p1 is equal to cp1; if the determination result is no, execute step 5); if the determination result is yes, execute step 6);

[0025] 5) Set the cp1 to p1 and return to step 2);

[0026] 6) adding 1 to the n;

[0027] 7) Determine whether n is greater than or equal to a preset number of times N, where N is greater than or equal to 2; if the determination result is no, return to step 3); if the determination result is yes, execute step 8);

[0028] 8) Output the reliable book page recognition result as p1.

[0029] Furthermore, when the program code is executed by the processor, it is also used to implement: after obtaining a reliable book page recognition result output, updating the current book page status recorded in the system.

[0030] Furthermore, when the program code is executed by the processor, it is also used to achieve: if the recognition result of the current page recognition is different from the previous N-1 consecutive page recognition results, then the current recognition result is not output or marked as unreliable.

[0031] Furthermore, when the program code is executed by the processor, it is also used to implement: if a finger is detected in the current image, and the finger is detected in the previous N-1 consecutive images, and the finger position (x, y) detected in the current image is the same as the finger position (x, y) detected in the previous N-1 consecutive images. i ,y i ) i＝1,…,N-1 The deviation between i =||(x, y)-(x i ,y i )|| are both less than a predetermined fixed value DP, the finger detection result is given as: the finger is reliably detected and the finger position is (x, y); otherwise, the finger detection result is marked as no finger is detected or the detection result is unreliable; according to the state of the page, and the change in the state and position of the finger, the playback state of the multimedia at different positions of the corresponding book page is determined.

[0032] A computer-readable storage medium is also provided, on which a program code is stored. The program code can be executed by a processor to implement the above-mentioned method for improving the reliability of picture book point reading recognition results.

[0033] In this invention, in order to solve the problem of preventing recognition errors from affecting user experience, a method is proposed to use multiple (2 or more) consecutive recognition results to fuse the recognition results and output extremely high-precision results to the user. The actual test results prove the effectiveness of this method, and the recognition error rate can be less than one thousandth, which greatly improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of outputting reliable book page recognition results according to the present invention;

[0035] Figure 2 It is a flow chart of outputting reliable finger detection results of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The specific implementation of the method and device for improving the reliability of picture book point reading recognition results in the terminal of the present invention is as follows:

[0038] After turning on the picture book reading function, the software system will take a picture of the book page (which may contain fingers) through the camera at regular intervals or after the last recognition is completed, and perform book page recognition, as well as finger presence and finger position detection on this image:

[0039] If the recognition result of this page is the same as the previous N-1 consecutive (N is greater than or equal to 2) page recognition results, the page recognition result is marked as reliable and output, otherwise the recognition result is not output or marked as unreliable. After obtaining a reliable page recognition result output, the software system updates the current page status recorded in the system.

[0040] If a finger is detected in this image, and the finger was detected in the previous N-1 consecutive images (N is greater than or equal to 2), and the finger position (x, y) detected this time is the same as the finger position (x, y) detected in the previous N-1 consecutive images (N is greater than or equal to 2), i ,y i ) i＝1,…,N-1 The deviation between i =||(x, y)-(x i ,y i )|| are both less than a pre-specified fixed value DP, the finger detection result is given as: the finger is reliably detected and the finger position is (x, y), otherwise the finger detection result is marked as no finger is detected or the detection result is unreliable. After obtaining a reliable finger detection result, the software system updates the current finger status and position recorded in the system.

[0041] The software system determines the playing status of multimedia at different positions of the corresponding book pages according to the book page status in the system and the changes in the finger status and position.

[0042] Reference Figure 1 , the method for improving the reliability of picture book point reading recognition results in the terminal comprises the steps of: 1) turning on the book point reading function and setting the initial value of the page state cp1 to be empty;

[0043] 2) Counter initialization, set counter n = 1;

[0044] 3) Setting a timer interval ⊿t, and taking a book page image through a camera in the terminal at a time interval ⊿t or after the last recognition is completed, and performing book page recognition on the captured image, and detecting the presence and position of a finger in the captured image; wherein the book page corresponding to the recognition is recorded as p1;

[0045] 4) Compare the book page p1 with the page state cp1; determine whether p1 is equal to cp1; if the determination result is no, execute step 5); if the determination result is yes, execute step 6);

[0046] 5) Set the cp1 to p1 and return to step 2);

[0047] 6) adding 1 to the n;

[0048] 7) Determine whether n is greater than or equal to a preset number of times N, where N is greater than or equal to 2; if the determination result is no, return to step 3); if the determination result is yes, execute step 8);

[0049] 8) Output the reliable book page recognition result as p1.

[0050] The method further includes: after obtaining a reliable book page recognition result output, updating the current book page status recorded in the system.

[0051] Furthermore, if the recognition result of the current page is different from the previous N-1 consecutive page recognition results, the current recognition result is not output or is marked as unreliable.

[0052] The step 3) detects the presence and position of a finger in the captured image, specifically including: if a finger is detected in the current image, and the finger is detected in the previous N-1 consecutive images, and the finger position (x, y) detected this time is the same as the finger position (x, y) detected in the previous N-1 consecutive images. i ,y i ) i＝1,…,N-1 The deviation between i =||(x, y)-(x i ,y i )|| are both less than a pre-specified fixed value DP, the finger detection result is given as: the finger is reliably detected and the finger position is (x, y); otherwise, the finger detection result is marked as no finger is detected or the detection result is unreliable.

[0053] Furthermore, the playing status of the multimedia at different positions of the corresponding book page is determined according to the page status and the change in the finger status and position.

[0054] See also Figure 2 , the method for outputting reliable finger detection results in the terminal comprises the steps of: 1) turning on the book point reading function, setting the finger presence state cf to none, and setting the finger position cp to empty;

[0055] 2) Counter initialization, set counter n = 1;

[0056] 3) Set the timer interval ⊿t, and take a picture of the book page through the camera in the terminal at a certain interval ⊿t or after the last recognition is completed, and detect the presence of the finger as f and the finger position as p;

[0057] 4) Determine whether f is equal to cf and f is true; if the determination result is no, execute step 5); if the determination result is yes, execute step 6);

[0058] 5) Set cf to f, output f, and then return to step 2);

[0059] 6) Calculate the distance Δp between p and cp = ||p-cp||; then execute step 7);

[0060] 7) Determine whether the Δp is less than a pre-specified threshold DP; if the determination result is no, execute step 8); if the determination result is yes, execute step 9);

[0061] 8) Set cp to p, and then return to step 2);

[0062] 9) adding 1 to the n;

[0063] 10) Determine whether n is greater than or equal to a preset number of times N, where N is greater than or equal to 2; if the determination result is no, return to step 3); if the determination result is yes, execute step 11);

[0064] 11) Output reliable finger detection results f and p.

[0065] In addition, a device for improving the reliability of picture book point reading recognition results in a terminal is also provided, the device comprising a processor and a memory, the memory storing program codes, and when the program codes are executed by the processor, they are used to implement the following functions:

[0066] 1) Enable the book reading function and set the initial value of page status cp1 to empty;

[0067] 2) Counter initialization, set counter n = 1;

[0068] 3) Setting a timer interval ⊿t, and taking a book page image through a camera in the terminal at a time interval ⊿t or after the last recognition is completed, and performing book page recognition on the captured image, and detecting the presence and position of a finger in the captured image; wherein the book page corresponding to the recognition is recorded as p1;

[0069] 4) Compare the book page p1 with the page state cp1; determine whether p1 is equal to cp1; if the determination result is no, execute step 5); if the determination result is yes, execute step 6);

[0070] 5) Set the cp1 to p1 and return to step 2);

[0071] 6) adding 1 to the n;

[0072] 7) Determine whether n is greater than or equal to a preset number of times N, where N is greater than or equal to 2; if the determination result is no, return to step 3); if the determination result is yes, execute step 8);

[0073] 8) Output the reliable book page recognition result as p1.

[0074] Furthermore, when the program code is executed by the processor, it is also used to implement: after obtaining a reliable book page recognition result output, updating the current book page status recorded in the system.

[0075] Furthermore, when the program code is executed by the processor, it is also used to achieve: if the recognition result of the current page recognition is different from the previous N-1 consecutive page recognition results, then the current recognition result is not output or marked as unreliable.

[0076] Furthermore, when the program code is executed by the processor, it is also used to implement: if a finger is detected in the current image, and the finger is detected in the previous N-1 consecutive images, and the finger position (x, y) detected in the current image is the same as the finger position (x, y) detected in the previous N-1 consecutive images. i ,y i ) i＝1,…,N-1 The deviation between i =||(x, y)-(x i ,y i )|| are both less than a predetermined fixed value DP, the finger detection result is given as: the finger is reliably detected and the finger position is (x, y); otherwise, the finger detection result is marked as no finger is detected or the detection result is unreliable; according to the state of the page, and the change in the state and position of the finger, the playback state of the multimedia at different positions of the corresponding book page is determined.

[0077] Furthermore, when the program code is executed by the processor, it is also used to implement:

[0078] 1) Enable the book reading function, set the finger presence status cf to none, and set the finger position cp to empty;

[0079] 2) Counter initialization, set counter n = 1;

[0080] 3) Set the timer interval ⊿t, and take a picture of the book page through the camera in the terminal at a certain interval ⊿t or after the last recognition is completed, and detect the presence of the finger as f and the finger position as p;

[0081] 4) Determine whether f is equal to cf and f is true; if the determination result is no, execute step 5); if the determination result is yes, execute step 6);

[0082] 5) Set cf to f, output f, and then return to step 2);

[0083] 6) Calculate the distance Δp between p and cp = ||p-cp||; then execute step 7);

[0084] 7) Determine whether the Δp is less than a pre-specified threshold DP; if the determination result is no, execute step 8); if the determination result is yes, execute step 9);

[0085] 8) Set cp to p, and then return to step 2);

[0086] 9) adding 1 to the n;

[0087] 10) Determine whether n is greater than or equal to a preset number of times N, where N is greater than or equal to 2; if the determination result is no, return to step 3); if the determination result is yes, execute step 11);

[0088] 11) Output reliable finger detection results f and p.

[0089] A computer-readable storage medium is also provided, on which a program code is stored. The program code can be executed by a processor to implement the above-mentioned method for improving the reliability of picture book point reading recognition results.

[0090] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0091] The method and device provided in the embodiments of the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for improving the reliability of picture book point reading recognition results in a terminal, It is characterized in that The method comprises the steps of: 1) starting the book point reading function and setting the initial value of the page state cp1 to be empty; 2) Counter initialization, set counter n = 1; 3) Set a timer interval ⊿t, and take a book page image through the camera in the terminal at a time interval ⊿t or after the last recognition is completed, and perform book page recognition on the photographed book page image, and record the recognized book page as p1; at the same time, if a finger is detected in this image, and the finger is detected in the previous N-1 consecutive images, and the finger position (x, y) detected this time is the same as the finger position (x, y) detected in the previous N-1 consecutive images i ,y i ) i＝1,…,N-1 The deviation between i =||(x, y)-(x i ,y i )|| are both less than a pre-specified fixed value DP, then the finger detection result is given as: the finger is reliably detected and the finger position is (x, y), otherwise the finger detection result is marked as no finger is detected or the detection result is unreliable; after obtaining a reliable finger detection result, update the current finger status and position recorded in the system; 4) Compare the book page p1 with the page state cp1 to determine whether p1 is equal to cp1; if the determination result is no, execute step 5); if the determination result is yes, execute step 6); 5) Set the cp1 to p1 and return to step 2); 6) adding 1 to the n; 7) Determine whether n is greater than or equal to a preset number of times N, where N is greater than or equal to 2; if the determination result is no, return to step 3); if the determination result is yes, execute step 8); 8) Output the reliable book page recognition result as p1.

2. The method according to claim 1, It is characterized in that Also includes: After obtaining reliable book page recognition result output, the current book page status recorded in the system is updated.

3. The method according to claim 2, It is characterized in that If the recognition result of the current page is different from the previous N-1 consecutive page recognition results, the current recognition result is not output or is marked as unreliable.

4. The method according to claim 3, It is characterized in that The playing status of the multimedia at different positions of the corresponding book page is determined according to the page status and the change in the finger status and position.

5. A device for improving the reliability of picture book point reading recognition results in a terminal, It is characterized in that The device includes a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the program codes are used to implement the following functions: 1) Enable the book reading function and set the initial value of page status cp1 to empty; 2) Initialize the counter and set the counter n=1; 3) Setting a timer interval ⊿t, taking a book page image through a camera in the terminal at a time interval ⊿t or after the last recognition is completed, and performing book page recognition on the photographed book page image, and recording the corresponding book page as p1; at the same time, if a finger is detected in this image, and the finger is detected in the previous N-1 consecutive images, and the finger position (x, y) detected this time is the same as the finger position (x, y) detected in the previous N-1 consecutive images i ,y i ) i＝1,…,N-1 The deviation between i =||(x, y)-(x i ,y i )|| are both less than a pre-specified fixed value DP, then the finger detection result is given as: the finger is reliably detected and the finger position is (x, y), otherwise the finger detection result is marked as no finger is detected or the detection result is unreliable; after obtaining a reliable finger detection result, update the current finger status and position recorded in the system; 4) Compare the book page p1 with the page state cp1; determine whether p1 is equal to cp1; if the determination result is no, execute step 5); if the determination result is yes, execute step 6); 5) Set the cp1 to p1 and return to step 2); 6) adding 1 to the n; 7) Determine whether n is greater than or equal to a preset number of times N, where N is greater than or equal to 2; if the determination result is no, return to step 3); if the determination result is yes, execute step 8); 8) Output the reliable book page recognition result as p1.

6. The device as claimed in claim 5, It is characterized in that When the program code is executed by the processor, it is also used to implement: after obtaining a reliable book page recognition result output, updating the current book page status recorded in the system.

7. The device according to claim 6, It is characterized in that When the program code is executed by the processor, it is also used to achieve: if the recognition result of the current page recognition is different from the previous N-1 consecutive page recognition results, then the current recognition result is not output or marked as unreliable.

8. The device according to claim 7, It is characterized in that When the program code is executed by the processor, it is also used to achieve: according to the state of the book page, and the change of the state and position of the finger, the playing state of the multimedia at different positions of the corresponding book page is determined.

9. A computer-readable storage medium, It is characterized in that The storage medium stores program codes, which can be executed by a processor to implement the method according to any one of claims 1 to 4.

Images