Image recognition-based reading assistance method, device, equipment and computer-readable medium

Abstract

The present invention discloses a reading assistance method, device, equipment and computer-readable medium based on image recognition, and relates to the field of computer technology. A specific implementation of the method includes: obtaining text information in an input image; analyzing the text information in the input image and a user behavior database to determine the predicted behavior of the input image, wherein the user behavior database includes text information and user behavior corresponding to the text information; and voice broadcasting the name of the predicted behavior. This implementation can convey the information in the image to the visually impaired, and then perform corresponding operations.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an image recognition-based reading assistance method, device, equipment and computer-readable medium. Background Art

[0002] At present, visually impaired people can be found everywhere. Visually impaired people generally have weak light perception, and the inconvenience of reading and writing seriously affects their daily life. The popularity of the Internet has changed their lives or fate to a certain extent. Through the Internet, visually impaired people can obtain a variety of information in the form of sound at any time.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: for the visually impaired, it is difficult to perform corresponding operations because they cannot obtain the information in the image. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide a reading assistance method, apparatus, device and computer-readable medium based on image recognition, which can transmit information in an image to the visually impaired, so that corresponding operations can be performed.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for assisting reading based on image recognition is provided, comprising:

[0006] Get the text information in the input image;

[0007] Determining a predicted behavior of the input image by analyzing text information in the input image and a user behavior database, wherein the user behavior database includes text information and user behavior corresponding to the text information;

[0008] The name of the predicted behavior is announced.

[0009] The input image is an image acquired in real time, or an image selected from stored images.

[0010] The step of analyzing the text information in the input image and the user behavior database to determine the predicted behavior of the input image includes:

[0011] The predicted behavior of the input image is determined by analyzing the text information in the input image and the user behavior database in combination with the Apriori algorithm.

[0012] Before analyzing the text information in the input image and the user behavior database to determine the predicted behavior of the input image, the method further includes:

[0013] The user behavior database is established according to the user information, the existing text information and the user behavior corresponding to the existing text information.

[0014] Before analyzing the text information in the input image and the user behavior database to determine the predicted behavior of the input image, the method further includes:

[0015] The user behavior database is established according to the user information, the existing text information, the user behavior corresponding to the existing text information and the usage environment information corresponding to the existing text information.

[0016] The existing text information includes the classification of the text information and the content of the text information.

[0017] The user behavior corresponding to the existing text information includes the classification of the user behavior, the content of the user behavior and the weight of the user behavior.

[0018] The user behavior database includes text information of historical input images of users and user behaviors corresponding to the text information of historical input images.

[0019] The step of analyzing the text information in the input image and the user behavior database to determine the predicted behavior of the input image includes:

[0020] According to the text information in the input image, a match is successfully made with the text information in the historical input image, and the user behavior corresponding to the text information in the historical input image is used as the predicted behavior of the input image.

[0021] According to a second aspect of an embodiment of the present invention, there is provided an image recognition-based reading assistance device, comprising:

[0022] A text module is used to obtain text information in an input image;

[0023] A behavior module, configured to determine a predicted behavior of the input image by analyzing the text information in the input image and a user behavior database, wherein the user behavior database includes text information and user behavior corresponding to the text information;

[0024] The announcement module is used to announce the name of the predicted behavior by voice.

[0025] According to a third aspect of an embodiment of the present invention, there is provided an image recognition-based reading-assisting electronic device, comprising:

[0026] one or more processors;

[0027] a storage device for storing one or more programs,

[0028] When the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0029] According to a fourth aspect of an embodiment of the present invention, there is provided a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the method as described above is implemented.

[0030] One embodiment of the above invention has the following advantages or beneficial effects: obtaining text information in an input image; determining the predicted behavior of the input image by analyzing the text information in the input image and a user behavior database, wherein the user behavior database includes text information and user behaviors corresponding to the text information; and announcing the name of the predicted behavior by voice. The predicted behavior is a behavior related to the text information in the image, so by announcing the predicted behavior, further operations of the user are facilitated.

[0031] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0033] Figure 1 is a schematic diagram of the main process of the image recognition-based reading assistance method according to an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of a scene for collecting images according to an embodiment of the present invention;

[0035] Figure 3 is a schematic diagram of a process for recognizing text information in an input image according to an embodiment of the present invention;

[0036] Figure 4 is a schematic diagram of establishing a user behavior database according to an embodiment of the present invention;

[0037] Figure 5 is another schematic diagram of establishing a user behavior database according to an embodiment of the present invention;

[0038] Figure 6 is a schematic diagram of the application process of the reading assistance method based on image recognition according to an embodiment of the present invention;

[0039] Figure 7 is a schematic diagram of the main structure of an image recognition-based reading assistance device according to an embodiment of the present invention;

[0040] Figure 8 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0041] Fig. 9 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0043] At present, according to the survey, 63% of the visually impaired people believe that the value of the Internet is very great, and the popularization of the Internet has changed their lives or destiny to a certain extent. In China, there are 130 million elderly people over 65 years old, more than 13 million visually impaired people, 20 million hearing impaired people, and 70 million people with dyslexia - a total of more than 20% of the Chinese population.

[0044] The majority of visually impaired people who participated in the survey were young and middle-aged people aged 12-50, accounting for more than 95% of the entire sample, among which those aged 21-30 and 31-40 accounted for 50% and 27% of the total respectively.

[0045] The disability level of the visually impaired is divided into four levels in the Practical Assessment Standards (Trial) Regulations for People with Disabilities in China. They are 1. First-level blindness (first-level visual disability); 2. Second-level blindness (second-level visual disability); 3. First-level low vision (third-level visual disability); 4. Second-level low vision (fourth-level visual disability). Among the visually impaired people surveyed, the vast majority are first-level blindness (first-level visual disability), accounting for 85%. The main characteristics of first-level blindness (first-level visual disability) are: no light perception (total blindness), weak light perception, corrected visual acuity less than 0.05, etc., which makes reading and writing very inconvenient and affects daily life.

[0046] For the visually impaired, it is difficult to perform corresponding operations because they cannot obtain the information in the image.

[0047] In order to solve the technical problem that the visually impaired cannot obtain the information in the image and thus have difficulty in performing corresponding operations, the following technical solution in the embodiments of the present invention may be adopted.

[0048] See also Figure 1 , Figure 1 Schematic diagram of the main process of the image recognition-based reading assistance method according to an embodiment of the present invention, inputting text information in an image, analyzing it in combination with a user behavior database to obtain the predicted behavior of the input image, and announcing the predicted behavior by voice. Figure 1 As shown, the specific steps include:

[0049] S101: Acquire text information in an input image.

[0050] There are many ways to obtain the text in the input image. In the embodiment of the present invention, the text in the input image can be obtained by optical character recognition technology.

[0051] Optical Character Recognition (OCR) refers to the process of electronic devices examining characters, determining their shapes by detecting dark and light patterns, and then translating the shapes into computer text using character recognition methods. That is, for printed characters, optical methods are used to convert the text in paper documents into black and white dot matrix image files, and recognition software is used to convert the text in the image into text format for further editing and processing by word processing software.

[0052] See also Figure 2 , Figure 2 4 is a schematic diagram of a scene for collecting images according to an embodiment of the present invention. Figure 2 The system includes users, terminals, and objects with text information. Users can send instructions to the terminal, and the terminal's camera takes a picture of the object to collect the image of the object. The terminal uses optical character recognition technology to obtain text information of the input image.

[0053] As an example, the object with text information can be a take-out business card with a phone number marked on it. As another example, the object with text information can be a promotional advertisement with a product name marked on it and posted on a wall.

[0054] It is understandable that the visually impaired have some light perception and can open the application (APP) in the terminal to control the terminal to collect images, including text information. After the terminal obtains the text information in the input image through optical character recognition technology, the terminal can voice broadcast the text information in the input image. In this way, the visually impaired can know the text information in the input image in a timely manner.

[0055] In one embodiment of the present invention, the input image may be an image captured in real time, or an image selected from stored images. Specifically, the visually impaired user, as a user, may capture images in real time through a terminal for recognition. In addition, the user may also select an image from stored images for recognition.

[0056] As an example, the user stores an image of a business card with a telephone number in a photo album of the terminal. When the telephone number needs to be used, the image can be selected from the photo album of the terminal for identification.

[0057] See also Figure 3 , Figure 31 is a schematic diagram of a process for recognizing text information in an input image according to an embodiment of the present invention, which specifically includes:

[0058] S301: Image input.

[0059] Capture images through the terminal's camera or select images from stored images.

[0060] S302: Preprocessing.

[0061] The outline of text information is obtained through preprocessing methods such as binarization, denoising, and tilt correction.

[0062] S303. Version analysis.

[0063] The outline of the text information is obtained by preprocessing, and then the layout is processed by segmentation and line division.

[0064] S304: character cutting.

[0065] Since Chinese characters belong to square characters and have radicals, we need to locate and segment the characters. We need to locate the boundary of the character string, segment the character string individually, and recognize the separated characters.

[0066] S305: Character feature extraction.

[0067] Extract and store the features of the characters, train a complete character feature template library, and provide a comparison database for subsequent recognition. The character feature template library can be the chi_sim.traineddata language package that has been trained based on the open source tool Tesseract.

[0068] S306: Character recognition.

[0069] The features extracted from the current character are compared with the trained character feature template library to identify the character. As an example, the features extracted from the current character can be filtered in the trained character feature template library to identify the character.

[0070] S307, document correction.

[0071] Correct the recognized characters for misspellings and polyphones based on the language context.

[0072] As an example, S306 and S307 may be performed based on the open source tool Tesseract.

[0073] exist Figure 3 In the embodiment of the present invention, the text information in the input image can be obtained by using OCR technology.

[0074] S102 . Analyze the text information in the input image and a user behavior database to determine the predicted behavior of the input image, wherein the user behavior database includes the text information and the user behavior corresponding to the text information.

[0075] The user behavior database includes text information and user behaviors corresponding to the text information. It can be considered that the user behavior database is the basis for determining the predicted behavior.

[0076] See also Figure 4 , Figure 4 It is a schematic diagram of establishing a user behavior database according to an embodiment of the present invention. In one embodiment of the present invention, a user behavior database can be established according to user information, existing text information and user behavior corresponding to the existing text information.

[0077] User information is used to draw user portraits. The same or similar user portraits have similar text information and user behaviors. When a user logs into the APP for the first time, the user can be guided to fill in the user information. As an example, user information includes one or more of the following: age, region, gender, and job.

[0078] The existing text information is the text information in the historical image. The user behavior corresponding to the existing text information is the user behavior corresponding to the text information in the historical image.

[0079] As an example, the existing text information includes a telephone number 123456. In the past, when the text information in the input image is the telephone number 123456, the user behavior of inputting the text in the image is to make a call. Therefore, the user behavior corresponding to the existing text information is: making a call.

[0080] It can be seen that the existing text information and the user behavior corresponding to the existing text information are corresponding to each other. The corresponding relationship between the existing text information and the user behavior corresponding to the existing text information can be a one-to-one corresponding relationship or a one-to-many relationship.

[0081] User information can identify user characteristics, and existing text information and user behavior corresponding to the existing text information can identify the historical correspondence between text information and user behavior. By establishing a user behavior database based on user information, existing text information, and user behavior corresponding to the existing text information, historical data can be effectively used to establish a user behavior database, which has great reference value.

[0082] In the embodiments of the present invention, the text information and the user behavior can be described from different perspectives, and the text information and the user behavior can be described in detail, so as to accurately describe the predicted behavior of the input image.

[0083] As an example, the existing text information includes the classification of text information and the content of text information. As an example, the classification of text information may include one or more of the following: text, numbers, special symbols and links. The content of text information includes characters.

[0084] As an example, the user behavior corresponding to the existing text information includes the classification of the user behavior, the content of the user behavior and the weight of the user behavior.

[0085] User behaviors can be classified into interactive user behaviors and autonomous user behaviors according to whether they require interaction. As an example, interactive user behaviors include user behaviors with clear directions, such as making calls and sending messages. As another example, autonomous user behaviors include user behaviors that do not require interaction, such as copying and pasting text and opening links.

[0086] The content of the user behavior includes the name of the user behavior and the subsequent execution direction of the user behavior. As an example, the name of the user behavior is: make a call, and the candidate execution mode of the user behavior is: call the call interface to enter the call making page.

[0087] The weight of user behavior includes the weight of each type of user behavior. User behavior can be divided into multiple categories in different ways, and a weight can be preset for each type of user behavior. As an example, user behavior can be divided into interactive user behavior and autonomous user behavior. The weight of interactive user behavior is preset to 0.6, and the weight of autonomous user behavior is preset to 0.4.

[0088] Figure 4 In the embodiment, in the user behavior database, the corresponding relationship between user information, existing text information and user behavior corresponding to the existing text information can be obtained, providing a basis for accurately determining the predicted behavior of the input image.

[0089] See also Figure 5 , Figure 5 This is another schematic diagram of establishing a user behavior database according to an embodiment of the present invention. In one embodiment of the present invention, a user behavior database can be established according to user information, existing text information, user behavior corresponding to the existing text information, and usage environment information corresponding to the existing text information.

[0090] exist Figure 4 On the basis of the embodiment, the user behavior database can also be established by combining the usage environment information corresponding to the existing text information.

[0091] The usage environment information corresponding to the existing text information is the usage environment information corresponding to the use of the existing text information. As an example, the usage environment information corresponding to the existing text information includes the use time and / or the use location.

[0092] As an example, if the existing text information is a phone number 123456, the user behavior corresponding to the existing text information is making a call, the time of making the call is 12 noon, and the location of making the call is a financial building. Then, combined with the usage environment information corresponding to the existing text information, it can be known that the phone number 123456 is likely to be a phone number for ordering food.

[0093] exist Figure 5 In the embodiment, a user behavior database is established by combining the usage environment information corresponding to the existing text information, so that the relationship between the environment information and the user behavior can be accurately known.

[0094] In one embodiment of the present invention, for each user, when using the APP for the first time, the predicted behavior of the input image is displayed and voice broadcast by inputting the text information in the image, and the user behavior database used includes text information based on other users and user behaviors corresponding to the text information.

[0095] With repeated use of the embodiments of the present invention, a user behavior database may be established for the user in consideration of the user's individual needs, and then the predicted behavior of the input image may be determined based on the user's user behavior database, which helps to improve the accuracy of the predicted behavior.

[0096] Specifically, the text information of the user's historical input images and the user behavior corresponding to the text information of the historical input images are uploaded, and then a user behavior database is established. It is understandable that the user behavior database can be stored on the terminal side or the network side. In the case where the user behavior database is stored on the network side, the user can obtain the user behavior database of the user from the network side after logging into the APP.

[0097] Then, based on the text information in the input image, after successfully matching the text information in the historical input image, the user behavior corresponding to the text information in the historical input image is used as the predicted behavior of the input image.

[0098] Since the user behavior database is established for the user himself, the predicted behavior of the input image is determined based on the user behavior database, which greatly improves the accuracy of the predicted behavior.

[0099] In an embodiment of the present invention, after determining the user behavior database, the predicted behavior of the input image can be determined based on the text information in the input image and the user behavior database in combination with data mining algorithm analysis.

[0100] As an example, data mining algorithms include C4.5 algorithm, k-nearest neighbor classification (KNN), Decision Tree Model, Classification And Regression Tree (CART) algorithm, k-means clustering, Apriori algorithm, Support Vector Machine (SVM), and Expectation–Maximization Algorithm (EM), etc.

[0101] In the embodiment of the present invention, based on the text information in the input image and the user behavior database, combined with the Apriori algorithm for analysis, the predicted behavior of the input image is obtained.

[0102] The Apriori algorithm is briefly introduced below. Apriori uses an iterative method called level-wise search, where k-itemsets are used to explore (k + 1)-itemsets. First, find the set of frequent 1-itemsets. This set is denoted as L1. L1 is used to find the set of frequent 2-itemsets L2, and L2 is used to find L3, and so on, until no frequent k-itemsets can be found. Finding each Lk requires one scan of the database.

[0103] The first step: In the user behavior database, establish the connection between user behaviors, simply referred to as the connection step.

[0104] To find Lk, generate the set of candidate k-itemsets by joining Lk-1 with itself. This set of candidate itemsets is denoted as Ck. Let l1 and l2 be the itemsets in Lk-1. The notation li[j] represents the j-th item of li. For example, l1[k - 2] represents the penultimate third item of l1.

[0105] Assume that the items in the transaction or itemset are sorted in lexicographical order. Perform the join of Lk-1 with Lk-1; among them, the elements of Lk-1 are joinable. If their first (k - 2) items are the same; that is, the elements l1 and l2 of Lk-1 are joinable if (l1[1] = l2[1]) ∧ (l1[2] = l2[2]) ∧ … ∧ (l1[k - 2] = l2[k - 2]) ∧ (l1[k - 1] < l2[k - 1]). The condition (l1[k - 1] < l2[k - 1])) simply ensures that no duplicates are generated. The resulting itemset from joining l1 and l2 is l1[1]l1[2]…l1[k - 1]l2[k - 1].

[0106] The second step: Confirm and filter out infrequent behaviors through the pruning step.

[0107] Pruning step: Ck is a superset of Lk; that is, its members may or may not be frequent, but all frequent k-itemsets are contained in Ck. Scan the user behavior database and determine the count of each candidate in Ck to determine Lk. By definition, all candidates with a count value not less than the minimum support count are frequent and thus belong to Lk. However, Ck can be large, so the amount of computation involved is large.

[0108] To compress Ck, we can use the Apriori property in the following way: any non-frequent (k-1)-itemset is not a subset of a frequent k-itemset. Therefore, if the (k-1)-subset of a candidate k-itemset is not in Lk-1, then the candidate is also unlikely to be frequent and can be removed from Ck. This subset test can be done quickly using a hash tree of all frequent itemsets.

[0109] Step 3: Generate association rules from frequent item sets

[0110] Once frequent itemsets are found from transactions in the user behavior database, it is straightforward to generate strong association rules from them. Strong association rules satisfy the minimum support and minimum confidence. For confidence, we can use formula (1), where the conditional probability is expressed by the item set support count.

[0111] confidence(A→B)=P(A│B)=support(A∪B) / support(A) (1)

[0112] Among them, support(A∪B) is the support count of (A∪B), and support(A) is the support count of A.

[0113] According to formula (1), the association rules can be generated as follows:

[0114] f1. For each frequent item set l, generate all non-empty subsets of l.

[0115] f2. For each non-empty subset s of l, if support(l) / support(s)≥min_conf, then output the rule Where min_conf is the minimum confidence threshold.

[0116] Since rules are generated from frequent itemsets, each rule automatically satisfies the minimum support. Frequent itemsets along with their support are pre-stored in a hash table so that they can be accessed quickly.

[0117] Step 4: Recommend predicted behavior.

[0118] S103: Voice announce the name of the predicted behavior.

[0119] In order to assist the visually impaired to perform corresponding operations, the name of the predicted behavior can be announced by voice. After the visually impaired knows the name of the predicted behavior, he can select the predicted behavior he needs.

[0120] In the above-mentioned embodiment of the present invention, the text information in the input image is obtained; the predicted behavior of the input image is determined based on the text information in the input image and the user behavior database, and the user behavior database includes the text information and the user behavior corresponding to the text information; the name of the predicted behavior is announced by voice. The predicted behavior is a behavior related to the text information in the image, so by announcing the predicted behavior, it is convenient for the user to further operate.

[0121] See also Figure 6 , Figure 6 1 is a schematic diagram of an application flow of an image recognition-based reading assistance method according to an embodiment of the present invention, which specifically includes:

[0122] S601: The camera collects input images.

[0123] The user opens the APP and calls the terminal's camera to capture an input image. As an example, the input image may be a business card hanging on a wall, which includes a phone number.

[0124] S602: Obtain text information in the input image through optical character recognition technology.

[0125] Through optical character recognition technology, the text information of the input image is obtained, that is, the telephone number in the business card: 234567.

[0126] S603: Voice broadcast the text information in the input image.

[0127] In order to facilitate the user to obtain the text information in the input image in time, the text information in the input image can be broadcasted by voice, that is, the telephone number in the photo: 234567.

[0128] S604: Determine the predicted behavior of the input image.

[0129] While executing S604, the predicted behavior of the input image may be determined. In an embodiment of the present invention, the predicted behavior of the input image may be determined by the terminal or by a server to which the terminal is connected. Exemplarily, the predicted behavior includes making a call, sending a text message, and searching for a phone number.

[0130] S605: Voice announce the name of the predicted behavior.

[0131] After determining the predicted behavior of the input image, the user may be informed of the name of the predicted behavior by voice announcement. For example, the voice announcement includes: making a call, sending a text message, and searching for a phone number.

[0132] Based on the voice broadcast of the above predicted behavior, the user can select the predicted behavior: dial a phone call. The terminal further calls the dialing interface to dial the phone number: 234567.

[0133] See also Figure 7 , Figure 7 is a schematic diagram of the main structure of an image recognition-based reading assistance device according to an embodiment of the present invention. The image recognition-based reading assistance device can implement an image recognition-based reading assistance method, such as Figure 7 As shown, the image recognition-based reading aid device specifically includes:

[0134] The text module 701 is used to obtain text information in the input image.

[0135] The behavior module 702 is used to analyze the text information in the input image and the user behavior database to determine the predicted behavior of the input image. The user behavior database includes the text information and the user behavior corresponding to the text information.

[0136] The announcement module 703 is used to announce the name of the predicted behavior by voice.

[0137] In one embodiment of the present invention, the input image is an image acquired in real time, or an image selected from stored images.

[0138] In one embodiment of the present invention, the behavior module 702 is specifically used to analyze the text information in the input image and the user behavior database in combination with the Apriori algorithm to determine the predicted behavior of the input image.

[0139] In one embodiment of the present invention, the behavior module 702 is specifically used to establish a user behavior database according to user information, existing text information and user behavior corresponding to the existing text information.

[0140] In one embodiment of the present invention, the behavior module 702 is specifically used to establish a user behavior database according to user information, existing text information, user behavior corresponding to the existing text information, and usage environment information corresponding to the existing text information.

[0141] In one embodiment of the present invention, the existing text information includes the classification of the text information and the content of the text information.

[0142] The user behavior corresponding to the existing text information includes the classification of the user behavior, the content of the user behavior and the weight of the user behavior.

[0143] In one embodiment of the present invention, the user behavior database includes text information of historical input images of users and user behaviors corresponding to the text information of the historical input images.

[0144] The behavior module 702 is specifically configured to successfully match the text information in the historical input image based on the text information in the input image, and use the user behavior corresponding to the text information in the historical input image as the predicted behavior of the input image.

[0145] Figure 8 An exemplary system architecture 800 is shown to which an image recognition-based assisted reading method or an image recognition-based assisted reading device according to an embodiment of the present invention can be applied.

[0146] like Figure 8 As shown, system architecture 800 may include terminal devices 801, 802, 803, a network 804, and a server 805. Network 804 is used to provide a medium for communication links between terminal devices 801, 802, 803 and server 805. Network 804 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0147] Users can use terminal devices 801, 802, and 803 to interact with server 805 through network 804 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 801, 802, and 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0148] The terminal devices 801 , 802 , and 803 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0149] The server 805 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 801, 802, and 803. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0150] It should be noted that the image recognition-based assisted reading method provided in the embodiment of the present invention is generally executed by the server 805 , and accordingly, the image recognition-based assisted reading device is generally disposed in the server 805 .

[0151] It should be understood that Figure 8The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0152] Reference below Fig. 9 , which shows a schematic diagram of the structure of a computer system 900 of a terminal device suitable for implementing an embodiment of the present invention. Fig. 9 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0153] like Fig. 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the system 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0154] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0155] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-mentioned functions defined in the system of the present invention are executed.

[0156] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0157] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0158] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as follows: a processor includes a sending unit, an acquisition unit, a determination unit, and a first processing unit. The names of these units do not, in some cases, constitute limitations on the units themselves. For example, the sending unit may also be described as a "unit that sends a request for obtaining an image to a connected server."

[0159] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0160] Get the text information in the input image;

[0161] Determining a predicted behavior of the input image by analyzing text information in the input image and a user behavior database, wherein the user behavior database includes text information and user behavior corresponding to the text information;

[0162] The name of the predicted behavior is announced.

[0163] According to the technical solution of the embodiment of the present invention, the text information in the input image is obtained; the predicted behavior of the input image is determined based on the text information in the input image and the user behavior database, and the user behavior database includes the text information and the user behavior corresponding to the text information; the name of the predicted behavior is voice broadcast. The predicted behavior is a behavior related to the text information in the image, so by broadcasting the predicted behavior, it is convenient for the user to further operate.

[0164] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A reading assistance method based on image recognition, characterized in that: include: Get the text information in the input image; The user behavior database includes text information of the user's historical input images and the user behavior corresponding to the text information of the historical input images. According to the text information in the input image, if the text information in the historical input image is successfully matched, the user behavior corresponding to the text information of the historical input image is used as the predicted behavior of the input image; Voice broadcasting the name of the predicted behavior; Before analyzing the text information in the input image and the user behavior database to determine the predicted behavior of the input image, the method further includes: Establishing the user behavior database according to the user information, the existing text information and the user behavior corresponding to the existing text information; the correspondence between the existing text information and the user behavior corresponding to the existing text information is a one-to-one correspondence or a one-to-many relationship; The user behavior corresponding to the existing text information includes the classification of the user behavior, the content of the user behavior and the weight of the user behavior; The classification of user behavior is divided into interactive user behavior and autonomous user behavior; The content of the user behavior includes the name of the user behavior and the subsequent execution direction of the user behavior.

2. The image recognition-based reading assistance method according to claim 1, characterized in that: The input image is an image acquired in real time, or an image selected from stored images.

3. The image recognition-based reading assistance method according to claim 1, characterized in that: The step of analyzing the text information in the input image and the user behavior database to determine the predicted behavior of the input image includes: The predicted behavior of the input image is determined by analyzing the text information in the input image and the user behavior database in combination with the Apriori algorithm.

4. The image recognition-based reading assistance method according to claim 1, characterized in that: Before analyzing the text information in the input image and the user behavior database to determine the predicted behavior of the input image, the method further includes: The user behavior database is established according to the user information, the existing text information, the user behavior corresponding to the existing text information and the usage environment information corresponding to the existing text information.

5. The image recognition-based reading assistance method according to claim 1 or 4, characterized in that: The existing text information includes the classification of text information and the content of text information.

6. A reading aid device based on image recognition, characterized in that: include: A text module is used to obtain text information in an input image; A behavior module, for a user behavior database including text information of a user's historical input image and user behaviors corresponding to the text information of the historical input image, and for a user behavior module, when a match is successfully made in the text information of the historical input image based on the text information in the input image, the user behaviors corresponding to the text information of the historical input image are used as predicted behaviors of the input image; and, establishing the user behavior database according to the user information, the existing text information and the user behavior corresponding to the existing text information; the correspondence between the existing text information and the user behavior corresponding to the existing text information is a one-to-one correspondence or a one-to-many relationship; The user behavior corresponding to the existing text information includes the classification of the user behavior, the content of the user behavior and the weight of the user behavior; The classification of user behaviors is divided into interactive user behaviors and autonomous user behaviors; the content of user behaviors includes the name of the user behaviors and the subsequent execution direction of the user behaviors; The announcement module is used to announce the name of the predicted behavior by voice.

7. An electronic device for assisting reading based on image recognition, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images