Image processing method and device
Through image processing technology, handwritten marking information on paper documents is identified and managed, which solves the problem of inefficient information management in the prior art, and realizes efficient query and storage of marking information on paper documents.
Patent Information
- Application Number
- CN202111310924.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-05
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2041-11-05
AI Technical Summary
The prior art is difficult to efficiently manage the marked information handwritten by users on paper documents, resulting in inefficient information management.
Through the image processing method, the target image of paper files is obtained, HSV information is extracted, pixel points with different colors than printed text are identified, mark information and mark types are identified, and handwritten mark information is realized to query and store handwritten mark information.
It realizes efficient management of labeled information on paper documents, improves information processing efficiency, and facilitates users to query and store handwritten content.
Smart Images

Figure CN114049639B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to an image processing method and device. Background Art
[0002] At present, when users read paper documents, they usually write some marking information on the paper documents. For example, they add marking information such as underlining to a certain content in the paper document to indicate that the content is the user's favorite content. Or, they write comments at a certain position in the paper document to indicate the user's opinion on the text at that position. After the user adds marking information to the paper document, how to manage this marking information is a problem that needs to be solved at present.
[0003] In the related art, after a user adds marking information to a paper document, the user usually manually copies the marking information into a special file for subsequent reference and use. This method of managing marking information is obviously very inefficient. Summary of the invention
[0004] The embodiment of the present application provides an image processing method and device, which can manage the annotations made by users on paper documents in an information-based manner. The technical solution is as follows:
[0005] In one aspect, an image processing method is provided, the method comprising:
[0006] Acquire a target image captured by an image acquisition device, wherein the target image is captured from a paper document;
[0007] Determine the hue, saturation and brightness (HSV) information of the target image;
[0008] Extracting pixels in the target image whose color is different from the color of the printed text based on the HSV information to obtain an intermediate image, wherein the intermediate image indicates an image displayed by the information marked by the user in the paper document;
[0009] Based on the intermediate image, identifying target marking information in the target image and a marking type of the target marking information, wherein the target marking information is information marked by a user in the paper document, and the marking type indicates a function that the user needs to implement through the target marking information;
[0010] Based on the marking type of the target marking information, the target content marked by the target marking information is queried or stored.
[0011] Optionally, the identifying, based on the intermediate image, target marking information and a marking type of the target marking information in the target image includes:
[0012] recognizing text information in the intermediate image based on an OCR algorithm;
[0013] If text information can be recognized from the intermediate image, the recognized text information is determined as the target mark information, the content indicated by the text information is determined as the target content, the mark type of the target mark information is determined to be handwritten text, and the target mark information indicates that the text information is information of text handwritten by the user in the paper document;
[0014] Based on the marking type of the target marking information, querying or storing the target content marked by the target marking information includes:
[0015] The target content is stored.
[0016] Optionally, the identifying, based on the intermediate image, target marking information and a marking type of the target marking information in the target image includes:
[0017] Matching the graphic indicated by the intermediate image with one or more graphic templates in a graphic template library, wherein each graphic template in the graphic template library corresponds to a marking type;
[0018] If the graphic indicated by the intermediate image matches the first graphic module, the graphic indicated by the intermediate image is determined as the target marking information, and the marking type corresponding to the first graphic template is determined as the marking type of the target marking information, and the first graphic template is any graphic template in the graphic template library.
[0019] Optionally, based on the marking type of the target marking information, before querying or storing the target content marked by the target marking information, the method further includes:
[0020] Acquire a local image area in the target image corresponding to the target marking information;
[0021] The text information in the local image area is recognized based on an optical character recognition (OCR) algorithm to obtain the target content marked by the target marking information.
[0022] Optionally, acquiring a local image area in the target image corresponding to the target marking information includes:
[0023] Pixels in the target image that are located above the figure indicated by the intermediate image and within a reference distance from the figure indicated by the intermediate image are extracted to obtain the local image area.
[0024] Optionally, the querying or storing the target content marked by the target marking information based on the marking type of the target marking information includes:
[0025] In the case where the mark type of the target mark information is query, querying the associated information of the target content, and returning the associated information of the target content to the user terminal, so that the user terminal displays the associated information of the target content;
[0026] When the mark type of the target mark information is favorite, the target content is stored.
[0027] Optionally, the method further comprises:
[0028] A mark type configuration instruction is received, where the mark type configuration instruction carries the one or more graphic templates and the mark types respectively corresponding to the one or more graphic templates.
[0029] In another aspect, an image processing device is provided, the device comprising:
[0030] An acquisition module, used to acquire a target image acquired by an image acquisition device, wherein the target image is acquired from a paper document;
[0031] A determination module, used to determine the hue, saturation and brightness (HSV) information of the target image;
[0032] An extraction module, configured to extract pixels in the target image whose color is different from the color of the printed text based on the HSV information, and obtain an intermediate image, wherein the intermediate image indicates an image displayed by the information marked by the user in the paper document;
[0033] an identification module, configured to identify target marking information in the target image and a marking type of the target marking information based on the intermediate image, wherein the target marking information is information marked by a user in the paper document, and the marking type indicates a function that the user needs to implement through the target marking information;
[0034] The processing module is used to query or store the target content marked by the target marking information based on the marking type of the target marking information.
[0035] Optionally, the identification module is used to:
[0036] If the target content marked by the target marking information is the content to be queried, querying the associated information of the target content;
[0037] The associated information of the target content is returned to the user terminal, so that the user terminal displays the associated information of the target content.
[0038] Optionally, the processing module is used to:
[0039] recognizing text information in the intermediate image based on an OCR algorithm;
[0040] If text information can be recognized from the intermediate image, the recognized text information is determined as the target mark information, the content indicated by the text information is determined as the target content, the mark type of the target mark information is determined to be handwritten text, and the target mark information indicates that the text information is information of text handwritten by the user in the paper document;
[0041] Correspondingly, the processing module is used to store the target content.
[0042] Optionally, the identification module is used to:
[0043] Matching the graphic indicated by the intermediate image with one or more graphic templates in a graphic template library, wherein each graphic template in the graphic template library corresponds to a marking type;
[0044] If the graphic indicated by the intermediate image matches the first graphic module, the graphic indicated by the intermediate image is determined as the target marking information, and the marking type corresponding to the first graphic template is determined as the marking type of the target marking information, and the first graphic template is any graphic template in the graphic template library.
[0045] Optionally, the identification module is used to:
[0046] Acquire a local image area in the target image corresponding to the target marking information;
[0047] The text information in the local image area is recognized based on an optical character recognition (OCR) algorithm to obtain the target content marked by the target marking information.
[0048] Optionally, the identification module is used to:
[0049] Pixels in the target image that are located above the figure indicated by the intermediate image and within a reference distance from the figure indicated by the intermediate image are extracted to obtain the local image area.
[0050] Optionally, the processing module is used to:
[0051] In the case where the mark type of the target mark information is query, querying the associated information of the target content, and returning the associated information of the target content to the user terminal, so that the user terminal displays the associated information of the target content;
[0052] When the mark type of the target mark information is favorite, the target content is stored.
[0053] Optionally, the device further comprises:
[0054] The receiving module is used to receive a mark type configuration instruction, where the mark type configuration instruction carries the one or more graphic templates and the mark types corresponding to the one or more graphic templates respectively.
[0055] In another aspect, an image processing device is provided, the device comprising:
[0056] processor;
[0057] Wherein, the processor is configured to execute any step in the above-mentioned image processing method.
[0058] On the other hand, a computer-readable storage medium is provided, on which instructions are stored, and when the instructions are executed by a processor, any step in the above-mentioned method for managing tag information is implemented.
[0059] In another aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to perform any step in the above method for managing tag information.
[0060] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:
[0061] Based on the embodiment of the present application, during the process of a user reading a paper document, if the user marks some information in the paper document, the target image collected for the paper document can be obtained, and the target marking information and the marking type of the target marking information made by the user in the paper document can be identified by processing the target image. Since the marking type indicates the function that the user needs to implement through the target marking information, based on the marking type of the target marking information, the target content marked by the target marking information can be queried or stored to realize the processing of the information manually marked by the user. That is, the embodiment of the present application provides an image processing method, which can realize the recognition of the marking information of the user on the paper document, and then realize the management of the manually marked information. Compared with the method of transcribing the content marked by the marking information, the method provided by the embodiment of the present application is more efficient. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0063] Figure 1 is a schematic diagram of an image processing system architecture provided by an embodiment of the present application;
[0064] Figure 2 is a flow chart of an image processing method provided by an embodiment of the present application;
[0065] Figure 3 It is a schematic diagram of a process for processing marking information provided by an embodiment of the present application;
[0066] Figure 4 is another flowchart of processing marking information provided by an embodiment of the present application;
[0067] Figure 5 is another flowchart of processing marking information provided by an embodiment of the present application;
[0068] Figure 6 This is a schematic diagram of a process for managing tag information provided by an embodiment of the present application;
[0069] Figure 7 is a block diagram of an image processing device provided in an embodiment of the present application;
[0070] Figure 8 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;
[0071] Fig. 9 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION
[0072] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0073] Before explaining the embodiments of the present application in detail, the application scenarios involved in the embodiments of the present application are first explained.
[0074] In recent years, with the improvement of people's living standards, people have paid more and more attention to student education issues, and traditional paper book reading has also received more and more attention from schools and parents. Therefore, how to assist students in the paper book reading process and provide relevant tools to improve students' reading efficiency is an important research direction in the field of intelligent education research.
[0075] At present, there are more and more studies on auxiliary means in the reading process at home and abroad. For example, there are many reading assistance and reading note management methods. This type of method mainly provides relevant functions for e-books. This function includes marking the content of the e-book during the e-book reading process, or selecting relevant content on the e-book, or entering your own reading notes on the e-book. This type of method can meet the user's needs for note-taking and reading assistance for e-books. However, for student users, the reading assistance function that is only for e-books cannot meet the current needs of paper book reading.
[0076] In addition, for the sake of protecting eyesight and cultivating learning habits, schools and parents currently recommend paper book reading. However, since the paper book reading process cannot realize the informatization of the book content, it is difficult to realize auxiliary functions such as note-taking like e-books. For example, when users are reading paper books, they cannot directly search for unknown nouns, or they can only hand-copy the sentences they want to record, or it is difficult for users to collect and organize their own notes while reading paper books.
[0077] Based on the above scenario, an embodiment of the present application provides an image processing method, which provides a reading tool method for paper books that better meets the actual needs of student users.
[0078] It should be noted that the marking information involved in the embodiment of the present application specifically refers to information such as marking symbols hand-drawn by the user on the paper document. The marking information may include information such as underline, circle, handwritten text, etc. The marking information involved in the following will be explained in this regard.
[0079] The system architecture involved in the embodiments of the present application is explained below.
[0080] Figure 1 Schematic diagram of an image processing system architecture provided by an embodiment of the present application. Figure 1 As shown, the image processing system 100 includes a user end 101 and a backend 102 .
[0081] The user terminal 101 and the backend 102 are connected by wire or wireless means for communication. The user terminal 101 can be a terminal device such as a mobile phone, a tablet computer or a desktop computer held by the user. The backend 102 can be a device such as a server, which can be a stand-alone server or a clustered server.
[0082] Specifically, the user terminal 101 is used to obtain an image of a paper document read by the user. For the convenience of subsequent description, the image is referred to as a target image. The target image may be an image directly captured from the paper document. Optionally, the target image may also be obtained by clipping the aforementioned captured image. For example, the captured image of the paper document is clipped of the surrounding non-paper area to obtain the target image.
[0083] For example, a camera may be configured on the user terminal 101 to capture an image of a paper document read by the user based on the camera, thereby obtaining the aforementioned target image. Optionally, the user terminal 101 may also receive images captured for paper documents sent by other terminals, thereby obtaining the target image. The embodiment of the present application does not limit how the user terminal 101 obtains the images captured for paper documents.
[0084] After acquiring the target image, the user terminal 101 can send the target image to the background 102. The background 102 can perform information management on the marking information made by the user on the paper document based on the image processing method provided in the embodiment of the present application.
[0085] It should be noted that Figure 1 The system architecture shown is described by taking the user terminal 101 and the backend 102 as independent devices as an example. Optionally, the method provided in the embodiment of the present application can also be applied to a centralized terminal device, which has the functions of the user terminal 101 and the backend 102, and the embodiment of the present application does not limit this.
[0086] also, Figure 1 The background shown can also be called a cloud platform, etc.
[0087] The image processing method provided in the embodiment of the present application is explained in detail below.
[0088] Figure 2 : is a flow chart of an image processing method provided by an embodiment of the present application. Figure 2 As shown in FIG. 1 , the method includes the following steps. It should be noted that, Figure 1 From the system architecture shown, we can see that Figure 2 The method shown can be applied in the background, and optionally, can also be applied in a centralized terminal device. Figure 2 The following is an example of an application in the background.
[0089] Step 201: Acquire a target image captured by an image capture device, where the target image is captured from a paper document.
[0090] In a possible implementation, the user end can capture an image of the paper document based on a camera configured by the user end, and then send the captured image to the background, which can use the image as the target image.
[0091] For example, when a user is reading a paper book, the user may annotate the paper book with relevant marking symbols or notes. After that, the user may take a photo of the paper page with the marking information through an application installed on the mobile phone, and the mobile phone may upload the taken photo to the background.
[0092] Optionally, after receiving the image sent by the user terminal, the background can also pre-process the image, and then use the pre-processed image as the target image. The pre-processing includes but is not limited to brightness adjustment, contrast adjustment, invalid area cropping, etc. Among them, the invalid area in the image can specifically be the area in the image other than the paper page. The embodiment of the present application does not limit the specific implementation method of the pre-processing.
[0093] Step 202: Determine the hue, saturation and value (HSV) information of the target image.
[0094] In HSV information, saturation indicates the degree to which a color is close to a spectral color. The higher the saturation, the darker the color and the closer it is to a spectral color; the lower the saturation, the lighter the color and the closer it is to white. A saturation of 0 indicates pure white. Lightness determines the brightness of a color in a color space. The higher the lightness, the brighter the color. A lightness of 0 indicates pure black (the color is the darkest at this time). In other words, when the hue is constant, a decrease in saturation means adding white to the spectral color, and the proportion of the spectral color is also decreasing. A decrease in saturation to 0 means that the proportion of the spectral color is zero, causing the entire color to appear white. A decrease in lightness means adding black to the spectral color, and the proportion of the spectral color is also decreasing. A decrease in lightness to 0 means that the proportion of the spectral color is zero, causing the entire color to appear black.
[0095] Since the color displayed on paper documents is the color displayed by ordinary printed text, and the color displayed by the user's manually marked information is usually different from the color displayed by the printed text, based on this principle, the user's handwritten mark information can be distinguished from the HSV information of the target image. In order to be able to recognize the user's handwritten mark information from the target image, it is necessary to first determine the HSV information of the target image.
[0096] In a possible implementation, the target image may be converted into an RGB (red-green-blue) image, and then the HSV information of the target image may be determined based on the RGB information in the RGB image. The HSV information of the target image may be determined based on the RGB information in the RGB image by an RGB-HSV conversion algorithm, which is not limited in the embodiments of the present application.
[0097] Optionally, the HSV information of the target image may also be determined by other methods, which is also not limited in the embodiments of the present application.
[0098] Step 203: extracting pixels in the target image whose color is different from the color of the printed text based on the HSV information to obtain an intermediate image, where the intermediate image indicates an image displayed by the information marked by the user in the paper document.
[0099] After the HSV information of the target image is obtained based on step 202, an intermediate image that can indicate the information marked by the user in the paper document can be obtained through step 203. The intermediate image only includes the mark information handwritten by the user, and does not include the text information on the paper document, so that the mark information handwritten by the user can be recognized based on the intermediate image later.
[0100] It should be noted that step 202 and step 203 determine the intermediate image in the target image based on the HSV information. Optionally, in another possible implementation, the background can also identify the intermediate image in the target image based on the first learning network. Among them, the first learning network is a learning network for identifying those marking information in the image that are marking information marked by the user. That is, a first learning network that can recognize marking information marked by the user is pre-trained so as to extract the intermediate image from the target image according to the first learning network. The embodiment of the present application does not limit the training process of the first learning network, but only needs to ensure that the trained first learning network meets the above functions.
[0101] Step 204: Based on the intermediate image, identify the target marking information and the marking type of the target marking information in the target image. The target marking information is the information marked by the user in the paper document, and the marking type indicates the function that the user needs to achieve through the target marking information.
[0102] Users usually write handwritten marking information on paper documents in the following scenarios. One scenario is: when users are reading paper documents, when they encounter words or sentences that they do not know or understand, they will add marking information such as underlines to these words or sentences so that they can be queried later. Another scenario is: when users are reading paper documents, when they encounter words or sentences that they think are good, they will add marking information such as circles to these words or sentences so that they can be collected and studied later. Another scenario is: when users are reading paper documents, when they have their own insights on certain words or sentences, they will write down their insights on paper documents.
[0103] It can be seen that different marking information of the user may have different functions. Therefore, in the embodiment of the present application, while identifying the target marking information handwritten by the user on the paper document, it is also necessary to identify the marking type of the target marking information, so as to determine the function that the user needs to achieve through the target marking information based on the marking type of the target marking information, and then process the target content marked by the target marking information through the following step 205, which processing includes query or storage, etc.
[0104] In addition, based on the aforementioned scenario of handwritten mark information, it can be known that the target mark information handwritten by the user can be divided into two categories, one is handwritten text information, and the other is hand-drawn graphics and other information. The recognition process of these two types of mark information is explained in detail below through two implementation methods.
[0105] The first implementation method is to identify the text information in the intermediate image based on the OCR (optical character recognition) algorithm; if the text information can be identified from the intermediate image, the identified text information is determined as the target mark information, the content indicated by the text information is determined as the target content, the mark type of the target mark information is determined to be handwritten text, and the target mark information indicates that the text information is the information of the text handwritten by the user in the paper document. Accordingly, if the text information is not identified from the intermediate image, there is no need to perform the subsequent processing process.
[0106] If text information can be recognized from the intermediate image, it indicates that the user has handwritten text content on the paper document. Therefore, the mark type of the target mark information can be directly determined as handwritten text, and the text content corresponding to the recognized text information can be determined as the target content marked by the target mark information.
[0107] Among them, OCR specifically refers to the process of analyzing and identifying image files of text materials to obtain text and layout information. In other words, the text in the image is recognized and returned in the form of text. Therefore, based on the OCR background, the text information in the intermediate image can be recognized to facilitate subsequent processing of this text information.
[0108] The second implementation method: match the graphic indicated by the intermediate image with one or more graphic templates in the graphic template library, each graphic template in the graphic template library corresponds to a marking type; if the graphic indicated by the intermediate image matches the first graphic module, the graphic indicated by the intermediate image is determined as the target marking information, and the marking type corresponding to the first graphic template is determined as the marking type of the target marking information, and the first graphic template is any graphic template in the graphic template library.
[0109] Among them, matching the graphic indicated by the intermediate image with one or more graphic templates in the graphic template library specifically refers to: determining the similarity between the intermediate image and each image template, obtaining the similarity corresponding to each image template, and determining the image template whose similarity exceeds the similarity threshold as the aforementioned first image template.
[0110] In addition, after determining the mark type of the target mark information, it is also necessary to identify the target content to be marked by the target mark information, so as to facilitate the subsequent processing of the target content. For example, the implementation process can be: obtaining the local image area corresponding to the target mark information in the target image; recognizing the text information in the local image area based on the optical character recognition (OCR) algorithm to obtain the target content marked by the target mark information.
[0111] The local image region corresponding to the target marking information refers to: the image region where the target content marked by the target marking information in the target image is located.
[0112] For example, in a scenario where text is marked by non-closed lines such as hand-drawn underlines, the implementation process of obtaining the local image area corresponding to the target marking information in the target image can be: extracting the pixel points in the target image that are above the figure indicated by the intermediate image and within a reference distance from the figure indicated by the intermediate image to obtain the local image area. The aforementioned reference distance is a preset distance, which is the distance occupied by a normal line of text in the printed text.
[0113] Optionally, the pixel points located above and below the figure indicated by the intermediate image and within the reference distance from the figure indicated by the intermediate image in the target image may be extracted to obtain the local image area. In this scenario, the target marking information in the local image area is located in the middle position.
[0114] As another example, in a scenario where text is marked by closed graphics such as hand-drawn circles, the above-mentioned process of obtaining the local image area corresponding to the target marking information in the target image can be: extracting the pixel points in the target image that are located within the graphics indicated by the intermediate image to obtain the local image area.
[0115] One or more graphic templates in the graphic template library are preconfigured. In a possible implementation, the background may receive a tag type configuration instruction, which carries one or more graphic templates and tag types corresponding to the one or more graphic templates. For example, the graphic template includes a graphic template indicating that the marked content is content to be queried, and the graphic template also includes a graphic template indicating that the marked content is content to be collected, that is, the tag type includes query or collection.
[0116] Among them, the tag type configuration instruction can be triggered by the user at the user end. In other words, the user can set the graphic templates corresponding to the various tag information that he is accustomed to according to his own preferences. Specifically, the user end displays a tag information configuration interface, which includes multiple tag configuration options, each tag configuration option indicates the function of a tag information, that is, corresponds to a tag type. Through the tag configuration option, the user can configure what kind of graphic template corresponds to the tag information with this function. When the user end detects a configuration operation for a certain tag configuration option, a tag type configuration instruction is generated accordingly. The tag type configuration instruction carries the graphic template corresponding to the tag information configured by the user and the tag type corresponding to the graphic template.
[0117] Based on the above configuration, after the background recognizes the intermediate image, it can match the intermediate image with a preset graphic template to determine how to process the content marked by the user for the target marking information.
[0118] Specifically, in the case where a graphic template indicating that the marked content is the content to be queried and a graphic template indicating that the marked content is the content to be collected are preconfigured, if the target marking information and the graphic template indicating that the marked content is the content to be queried match, then it is determined that the target content marked by the target marking information is the content to be queried. Alternatively, if the target marking information and the graphic template indicating that the marked content is the content to be collected match, then it is determined that the target content marked by the target marking information is the content to be collected.
[0119] In the embodiment of the present application, in order to realize information management of various tag information, different types of tag information can be processed differently, so that users can quickly and efficiently query these tag information later. That is, after determining the tag type of the target tag information, the content marked by the target tag information can be processed through the following step 205.
[0120] Step 205: Based on the mark type of the target mark information, query or store the target content marked by the target mark information.
[0121] For the first implementation of step 204, the implementation process of step 205 is: storing the target content. Since the handwritten text is to some extent the content that the user wants to view later, after the target content handwritten by the user is recognized, the target content can be stored.
[0122] For the second implementation in step 204, in some embodiments, the implementation process of step 205 may be: when the mark type of the target mark information is query, query the associated information of the target content; return the associated information of the target content to the user terminal, so that the user terminal displays the associated information of the target content. That is, if the target mark information indicates that the marked content is the target content that the user needs to query, the background queries the associated information of the target content to feedback the associated information to the user.
[0123] Figure 3 1 is a flowchart of processing tag information provided by an embodiment of the present application. Assume that a tag information such as a "circle" is pre-configured to indicate that the marked content is the content to be queried by the user. Figure 3 As shown in the figure, when reading a paper book, the user marks a word "layer upon layer" on a certain page in a circle. The background obtains the image of the page ( Figure 3 When the target image is marked as a target image in the image, and the target marking information is identified as a "circle" based on the aforementioned step 204, the target content "layers and layers" marked by the marking information "circle" is obtained, and the associated information of the target content is queried, so that the user terminal displays the associated information of the target content. Figure 3 As shown, the associated information exemplarily includes the pinyin of the word, the explanation of the word, the source of the word, and other sentences including the word, so that the user can understand the common writing methods of the target content based on these other sentences and practice the target content according to the other sentences.
[0124] pass Figure 3As shown in the process of processing marking information, when students encounter words they do not understand while reading paper books, they can circle them on the paper books with a pen, and then use the mobile app to take a photo and upload it to the background. The background then obtains the specific content of the words circled by the students through image processing and OCR recognition. The background searches for the words and obtains the specific meaning of the words. At the same time, according to the content associated with the words, the system will provide related functions, including common methods of vocabulary, sentence making exercises, and if it is a proper noun, the meaning and specific content of the noun will be displayed in detail.
[0125] Regarding the second implementation manner in step 204, in some other embodiments, the implementation process of step 205 may be: when the mark type of the target mark information is favorite, storing the target content.
[0126] In combination with the aforementioned processing method for handwritten text, it can be seen that in an embodiment of the present application, if the content marked by the target marking information is the target content that the user wants to collect or the target content such as the user's handwritten text, the background will identify the target content and store it for subsequent user reference.
[0127] In addition, after storing the target content, the backend can also configure attributes for the stored target content to facilitate subsequent users to consult the stored target content, so as to facilitate subsequent query of the target content based on the attributes. Therefore, in a possible implementation, the backend can configure one or more attribute options for the target content, and any of the one or more attribute options indicates that the user configures an attribute of the target content. The attribute options can be, for example, subject, creation time, etc.
[0128] In this scenario, the user can configure the properties of the target content. Specifically, the backend responds to the configuration instruction for the first property option, obtains the first property for the target content, the first property option is any property option among one or more property options, and stores the first property for the target content.
[0129] Among them, the configuration instruction for the first attribute option is triggered by the user through a preset operation. For example, an attribute configuration interface is displayed on the user-side display interface. When the user enters an attribute on the interface, the user side can use the attribute entered by the user as an attribute of the target content and send the configuration instruction to the background so that the background can bind the attribute configured by the user and the target content for subsequent reference.
[0130] Optionally, the attributes of the stored target content can also be automatically generated by the background according to the target content, for example, the background can use the keywords in the target content as the attributes of the target content, or use the time when the target content is stored as the attributes of the target content. That is, the attributes of the target content can also be configured by the system default.
[0131] After configuring the attributes for each stored target content, if the background receives a query instruction later, and the query instruction carries one or more target attributes, the background can search for target content whose attributes meet one or more target attributes from the stored target content, obtain the query result, and then return the query result to the user end.
[0132] The target content whose attributes satisfy one or more target attributes may specifically mean that the attributes of the target content include all the attributes of the one or more target attributes. It may also mean that the attributes of the target content include some of the attributes of the one or more target attributes. This embodiment of the application does not limit this.
[0133] Figure 4 2 is another flowchart of processing mark information provided by an embodiment of the present application. Assume that mark information such as a "horizontal line" is pre-configured to indicate that the marked content is the content to be collected by the user. Figure 4 In the example, a user marked a sentence on a certain page with a horizontal line while reading a paper book: "I walked under the tree, and the sky sprinkled cherry blossom rain like elves." The background obtains the image for this page ( Figure 4 When the target image is marked in the image, and the target mark information is identified as a "horizontal line" based on OCR, the target content "I walked under the tree, and the sky sprinkled cherry blossom rain like elves" marked by the mark information "horizontal line" is obtained and stored. At the same time, the attributes of the target content are set based on user configuration or system default configuration, and the attributes include the theme of cherry blossoms and the time of 2021-6-30 for subsequent reference.
[0134] pass Figure 4 As shown in the process of processing marking information, when students read paper books and encounter sentences or paragraphs they like, they can draw horizontal lines to mark the content they want to collect, and then use the mobile app to take photos and upload them to the background. The background obtains the sentences and paragraphs collected by students through image processing and OCR recognition, and stores the recognized content. Users can configure attributes for their favorite content so that the background can classify and organize the collection based on these attributes. Later, users can search by time, content, and keywords, so that users can easily find the good sentences they collected before.
[0135] Figure 5 FIG. 1 is another flow chart of processing tag information provided by an embodiment of the present application. Figure 5 In the example, a user writes a sentence "a favorite paragraph" on a certain page while reading a paper book. The backend obtains the image of the page ( Figure 5When the target image is marked as a target image in the image, and the target mark information is identified as "handwritten text" based on OCR, the target content "favorite paragraph" marked by the mark information is obtained and stored. At the same time, the properties of the target content are set based on the user configuration or the system default configuration, and the properties include the theme of cherry blossoms, the time of 2021-6-30, and the page number of the handwritten text, etc., for subsequent reference.
[0136] based on Figure 5 As shown in the process of processing marking information, when students read paper books, they write down their feelings and insights on the paper books, and then use the mobile app to take photos and upload them to the background. The background obtains the content of the students' reading notes through image processing and OCR recognition, and stores the recognized reading notes. Users can configure attributes for their own reading notes so that the background can classify and organize the reading notes based on these attributes. Later, users can search by time, content, and keywords, so that users can easily find previous reading notes.
[0137] It can be seen that based on the method provided in the embodiment of the present application, when student users read paper books, they can mark or write notes on the content of the book, and then the students can use the mobile app to take pictures and upload them to the background. The background uses OCR recognition technology to obtain the student's specific marking type, including unknown vocabulary marks, favorite paragraphs, and handwritten note content. Depending on the type of student marking, the background determines the corresponding processing method and obtains relevant results. For example, the retrieved vocabulary information is directly returned to the student through the app; the identified favorite paragraphs and handwritten note content are stored, and retrieval and management functions are provided for students.
[0138] Figure 6 2 is a flowchart of managing tag information provided by an embodiment of the present application. The flowchart is used to illustrate the above-mentioned examples of steps 201 to 203.
[0139] like Figure 6 As shown, student users read paper books, make relevant marks and write notes during the reading process, and take photos through the mobile phone app and upload them to the background; the background obtains the type of student marks through OCR recognition, and performs different processing respectively: for vocabulary retrieval needs, perform retrieval operations and return vocabulary explanation results; for paragraph collection operations, the background stores the collected paragraph content and provides corresponding management functions; for handwritten note operations, the recognized note content is stored in the background and provides corresponding management functions.
[0140] It should be noted that Figure 6It is only used to illustrate the method for managing tag information provided in the embodiment of the present application, and does not constitute a limitation on the aforementioned steps 201 to 203.
[0141] In summary, in an embodiment of the present application, during the process of a user reading a paper document, if the user marks some information in the paper document, a target image collected for the paper document can be obtained, and by processing the target image, the target marking information and the marking type of the target marking information made by the user in the paper document can be identified. Since the marking type indicates the function that the user needs to implement through the target marking information, based on the marking type of the target marking information, the target content marked by the target marking information can be queried or stored to realize the processing of the information manually marked by the user. That is, an embodiment of the present application provides an image processing method, which can realize the recognition of the marking information on the paper document by the user, and then realize the management of the manually marked information. Compared with the method of transcribing the content marked by the marking information, the method provided by the embodiment of the present application is more efficient.
[0142] All the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and the embodiments of the present application will not be described in detail one by one.
[0143] Figure 7 is a block diagram of a device for managing tag information provided by an embodiment of the present application. The device can be implemented by software, hardware, or a combination of both. Figure 7 As shown, the device 700 may include the following modules.
[0144] An acquisition module 701 is used to acquire a target image acquired by an image acquisition device, where the target image is acquired from a paper document;
[0145] A determination module 702 is used to determine the hue, saturation and brightness (HSV) information of the target image;
[0146] An extraction module 703 is used to extract pixels in the target image whose color is different from the color of the printed text based on the HSV information, and obtain an intermediate image, where the intermediate image indicates an image displayed by the information marked by the user in the paper document;
[0147] The recognition module 704 is used to recognize the target marking information and the marking type of the target marking information in the target image based on the intermediate image, where the target marking information is the information marked by the user in the paper document, and the marking type indicates the function that the user needs to realize through the target marking information;
[0148] The processing module 705 is used to query or store the target content marked by the target marking information based on the marking type of the target marking information.
[0149] Optionally, the identification module is used to:
[0150] If the target marking information indicates that the marked target content is the content to be queried, querying the associated information of the target content;
[0151] The associated information of the target content is returned to the user terminal, so that the user terminal displays the associated information of the target content.
[0152] Optionally, the processing module is used to:
[0153] Recognize text information in the intermediate image based on OCR algorithm;
[0154] If text information can be recognized from the intermediate image, the recognized text information is determined as target mark information, the content indicated by the text information is determined as target content, the mark type of the target mark information is determined to be handwritten text, and the target mark information indicates that the text information is information that the user handwrites in a paper document;
[0155] Accordingly, the processing module is used to store the target content.
[0156] Optionally, the identification module is used to:
[0157] Matching the graphic indicated by the intermediate image with one or more graphic templates in a graphic template library, each graphic template in the graphic template library corresponding to a marking type;
[0158] If the graphic indicated by the intermediate image matches the first graphic module, the graphic indicated by the intermediate image is determined as the target marking information, and the marking type corresponding to the first graphic template is determined as the marking type of the target marking information. The first graphic template is any graphic template in the graphic template library.
[0159] Optionally, the identification module is used to:
[0160] Obtaining a local image area corresponding to the target marking information in the target image;
[0161] The text information in the local image area is recognized based on the optical character recognition (OCR) algorithm to obtain the target content marked by the target marking information.
[0162] Optionally, the identification module is used to:
[0163] Pixels in the target image that are located above the figure indicated by the intermediate image and within a reference distance from the figure indicated by the intermediate image are extracted to obtain a local image area.
[0164] Optionally, the processing module is used to:
[0165] In the case where the mark type of the target mark information is query, querying the associated information of the target content, and returning the associated information of the target content to the user terminal, so that the user terminal displays the associated information of the target content;
[0166] When the mark type of the target mark information is favorite, the target content is stored.
[0167] Optionally, the device further comprises:
[0168] The receiving module is used to receive a mark type configuration instruction, where the mark type configuration instruction carries one or more graphic templates and mark types corresponding to the one or more graphic templates.
[0169] In summary, in an embodiment of the present application, during the process of a user reading a paper document, if the user marks some information in the paper document, a target image collected for the paper document can be obtained, and by processing the target image, the target marking information and the marking type of the target marking information made by the user in the paper document can be identified. Since the marking type indicates the function that the user needs to implement through the target marking information, based on the marking type of the target marking information, the target content marked by the target marking information can be queried or stored to realize the processing of the information manually marked by the user. That is, an embodiment of the present application provides an image processing method, which can realize the recognition of the marking information on the paper document by the user, and then realize the management of the manually marked information. Compared with the method of transcribing the content marked by the marking information, the method provided by the embodiment of the present application is more efficient.
[0170] It should be noted that: the image processing device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when processing images. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the image processing device provided in the above embodiment and the image processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0171] Figure 8 FIG. 8 is a block diagram showing a structure of a terminal 800 provided by an exemplary embodiment of the present application. Figure 8The terminal 800 may be implemented by using the terminal shown in FIG. The terminal 800 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 800 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.
[0172] Typically, the terminal 800 includes a processor 801 and a memory 802 .
[0173] The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0174] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one instruction, which is used to be executed by the processor 801 to implement the image processing method provided in the method embodiment of the present application.
[0175] In some embodiments, the terminal 800 may further optionally include: a peripheral device interface 803 and at least one peripheral device. The processor 801, the memory 802 and the peripheral device interface 803 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 803 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, a positioning assembly 808 and a power supply 809.
[0176] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the terminal 800, and the terminal 800 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.
[0177] The embodiment of the present application also provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by the processor of the terminal, the terminal is enabled to execute the image processing method provided in the above embodiment.
[0178] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a terminal, enables the terminal to execute the image processing method provided in the above embodiment.
[0179] Fig. 9 is a schematic diagram of a server structure provided according to an embodiment of the present application. Fig. 9 The server shown in the figure is implemented, and the server can be a server in a background server cluster.
[0180] Specifically:
[0181] The server 900 includes a central processing unit (CPU) 901, a system memory 904 including a random access memory (RAM) 902 and a read-only memory (ROM) 903, and a system bus 905 connecting the system memory 904 and the central processing unit 901. The server 900 also includes a basic input / output system (I / O system) 906 that helps transfer information between various components in the computer, and a large-capacity storage device 907 for storing an operating system 913, application programs 914, and other program modules 915.
[0182] The basic input / output system 906 includes a display 908 for displaying information and an input device 909 such as a mouse and a keyboard for user inputting information. The display 908 and the input device 909 are connected to the central processing unit 901 through an input / output controller 910 connected to the system bus 905. The basic input / output system 906 may also include an input / output controller 910 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 910 also provides output to a display screen, a printer, or other types of output devices.
[0183] The mass storage device 907 is connected to the central processing unit 901 through a mass storage controller (not shown) connected to the system bus 905. The mass storage device 907 and its associated computer-readable media provide non-volatile storage for the server 900. That is, the mass storage device 907 may include a computer-readable medium (not shown) such as a hard disk or a CD-ROM drive.
[0184] Without loss of generality, computer readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 904 and mass storage device 907 can be collectively referred to as memory.
[0185] According to various embodiments of the present application, the server 900 can also be connected to a remote computer on the network through a network such as the Internet. That is, the server 900 can be connected to the network 912 through the network interface unit 911 connected to the system bus 905, or the network interface unit 911 can be used to connect to other types of networks or remote computer systems (not shown).
[0186] The memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU. The one or more programs include instructions for performing the image processing method provided in the embodiment of the present application.
[0187] The embodiment of the present application also provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by the processor of the server, the server can execute the image processing method provided by the above embodiment.
[0188] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a server, enables the server to execute the image processing method provided in the above embodiment.
[0189] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0190] The above description is only a preferred embodiment of the embodiments of the present application and is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.
Claims
1. An image processing method, characterized in that: The method comprises: Acquire a target image captured by an image acquisition device, wherein the target image is captured by a user for a paper document during reading of the paper document, wherein the user is allowed to annotate mark information on the paper document during reading of the paper document, wherein the mark information includes a mark symbol or a note; Determine the hue, saturation, and brightness (HSV) information of the target image; extract pixels in the target image whose color is different from the color of the printed text based on the HSV information to obtain an intermediate image, wherein the intermediate image indicates an image displayed by the information marked by the user in the paper document; In the case where text information exists in the intermediate image, the text information in the intermediate image is identified based on an OCR algorithm; if the text information can be identified from the intermediate image, the identified text information is determined as the target marking information, the content indicated by the text information is determined as the target content, the marking type of the target marking information is determined to be handwritten text, and the target marking information indicates that the text information is information of text handwritten by the user in the paper document; based on the marking type of the target marking information being handwritten text, the target content is stored; In the case where there is a graphic in the intermediate image, the graphic indicated by the intermediate image is matched with one or more graphic templates in the graphic template library, each graphic template in the graphic template library corresponds to a tag type, including tag types including query and collection; if the graphic indicated by the intermediate image matches the first graphic module, the graphic indicated by the intermediate image is determined as the target tag information, and the tag type corresponding to the first graphic template is determined as the tag type of the target tag information, and the first graphic template is any graphic template in the graphic template library; in the case where the tag type of the target tag information is query, the associated information of the target content marked by the target tag information is queried, and the associated information of the target content is returned to the user terminal, so that the user terminal displays the associated information of the target content; in the case where the tag type of the target tag information is collection, the target content marked by the target tag information is stored.
2. The method according to claim 1, characterized in that Based on the mark type of the target mark information, before querying or storing the target content marked by the target mark information, the method further includes: Acquire a local image area in the target image corresponding to the target marking information; The text information in the local image area is recognized based on an optical character recognition (OCR) algorithm to obtain the target content marked by the target marking information.
3. The method according to claim 2, characterized in that The obtaining of a local image area in the target image corresponding to the target marking information includes: Pixels in the target image that are located above the figure indicated by the intermediate image and within a reference distance from the figure indicated by the intermediate image are extracted to obtain the local image area.
4. The method according to claim 1, characterized in that The method further comprises: A mark type configuration instruction is received, where the mark type configuration instruction carries the one or more graphic templates and the mark types respectively corresponding to the one or more graphic templates.
5. An image processing device, characterized in that: The apparatus comprises a processor; The processor is used to: Acquire a target image captured by an image acquisition device, wherein the target image is captured by a user for a paper document during reading of the paper document, wherein the user is allowed to annotate mark information on the paper document during reading of the paper document, wherein the mark information includes a mark symbol or a note; Determine the hue, saturation, and brightness (HSV) information of the target image; extract pixels in the target image whose color is different from the color of the printed text based on the HSV information to obtain an intermediate image, wherein the intermediate image indicates an image displayed by the information marked by the user in the paper document; In the case where text information exists in the intermediate image, the text information in the intermediate image is identified based on an OCR algorithm; if the text information can be identified from the intermediate image, the identified text information is determined as the target marking information, the content indicated by the text information is determined as the target content, the marking type of the target marking information is determined to be handwritten text, and the target marking information indicates that the text information is information of text handwritten by the user in the paper document; based on the marking type of the target marking information being handwritten text, the target content is stored; In the case where there is a graphic in the intermediate image, the graphic indicated by the intermediate image is matched with one or more graphic templates in the graphic template library, each graphic template in the graphic template library corresponds to a tag type, including tag types including query and collection; if the graphic indicated by the intermediate image matches the first graphic module, the graphic indicated by the intermediate image is determined as the target tag information, and the tag type corresponding to the first graphic template is determined as the tag type of the target tag information, and the first graphic template is any graphic template in the graphic template library; in the case where the tag type of the target tag information is query, the associated information of the target content marked by the target tag information is queried, and the associated information of the target content is returned to the user terminal, so that the user terminal displays the associated information of the target content; in the case where the tag type of the target tag information is collection, the target content marked by the target tag information is stored.
