Image processing method and device, electronic device and storage medium

Abstract

An image processing method, an image processing device, an electronic device and a storage medium. The image processing method comprises: obtaining an image to be identified; identifying the image to be identified to obtain a plurality of area frames, a plurality of area information corresponding to the plurality of area frames and a plurality of contents to be typeset; typeset the plurality of contents to be typeset based on the image to be identified and the plurality of area information to obtain a typeset document corresponding to the image to be identified. The image processing method can obtain an electronic document corresponding to the image to be identified, thereby facilitating various operations on the image to be identified, such as storage, transmission, management, backup, printing and the like.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to an image processing method, an image processing apparatus, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0002] Users can take photos of paper documents to extract relevant information from them and electronically archive them, so as to facilitate the management and storage of paper documents. For example, in some usage scenarios, students will use a large number of test papers, homework, exercise books, etc. in the learning process, and need to organize and repeatedly practice these test papers, homework, and exercise books. Efficient and convenient test paper management, test paper storage, wrong question recording, etc. can significantly improve students' learning efficiency; for example, in other usage scenarios, with the development of mobile networks, students often study online and submit homework through the Internet, so they need to obtain homework documents with clear backgrounds to submit through the Internet. Summary of the invention

[0003] At least one embodiment of the present disclosure provides an image processing method, comprising: acquiring an image to be identified; identifying the image to be identified to obtain a plurality of area frames, a plurality of area information corresponding one by one to the plurality of area frames, and a plurality of contents to be typeset; and typeset the plurality of contents to be typeset based on the image to be identified and the plurality of area information to obtain a typeset document corresponding to the image to be identified.

[0004] For example, in the image processing method provided by at least one embodiment of the present disclosure, the image to be identified is identified to obtain multiple area boxes, multiple area information corresponding one-to-one to the multiple area boxes, and multiple contents to be typeset, including: identifying the image to be identified through an object detection model to obtain the multiple area boxes and the multiple area information, wherein the multiple area boxes include multiple first text boxes; identifying the multiple first text boxes through a text recognition model to obtain multiple text contents corresponding one-to-one to the multiple first text boxes; wherein the multiple contents to be typeset include one or more of the multiple text contents.

[0005] For example, the image processing method provided by at least one embodiment of the present disclosure also includes: determining a plurality of text categories corresponding one to one to the plurality of first text boxes based on the plurality of area information and the plurality of text contents, wherein the area information corresponding to any first text box among the plurality of first text boxes includes the text category of the any first text box.

[0006] For example, in the image processing method provided by at least one embodiment of the present disclosure, the multiple area boxes also include at least one picture frame, and the image to be identified is identified to obtain multiple area boxes, multiple area information corresponding to the multiple area boxes one by one, and multiple contents to be typeset, and also includes: extracting at least one picture to be typeset corresponding to the at least one picture frame, wherein the multiple contents to be typeset also include the at least one picture to be typeset.

[0007] For example, in the image processing method provided by at least one embodiment of the present disclosure, the image to be identified is an image containing at least one title, the multiple area boxes also include at least one title box corresponding one-to-one to the at least one title, each title box includes at least one first text box in the area covered by the image to be identified, each area information includes position information of the area box corresponding to each area information in the image to be identified, and based on the multiple area information and the multiple text contents, multiple text categories corresponding one-to-one to the multiple first text boxes are determined, including: determining the correspondence between the at least one title box and the multiple first text boxes based on the position information in the multiple area information; and determining the multiple text categories based on the correspondence and the multiple text contents.

[0008] For example, in the image processing method provided by at least one embodiment of the present disclosure, the at least one title box includes a first title box, in a first direction, the first title box has a first edge, the multiple first text boxes include a first to-be-processed box, the multiple text categories include a major question name, and based on the corresponding relationship and the multiple text contents, the multiple text categories are determined, including: in response to the corresponding relationship indicating that the first to-be-processed box is located in the area covered by the first title box in the to-be-recognized image, and there is no area box between the first to-be-processed box and the first edge, determining that the text category of the first to-be-processed box is the major question name; or, in response to the corresponding relationship indicating that the first to-be-processed box is located outside the area covered by the first title box in the to-be-recognized image, and there is no area box between the first to-be-processed box and the first edge, and determining that the text content corresponding to the first to-be-processed box contains major question feature information, determining that the text category of the first to-be-processed box is the major question name.

[0009] For example, in the image processing method provided by at least one embodiment of the present disclosure, the multiple first text boxes include a second box to be processed, the multiple text categories include a title, in a first direction, the second box to be processed has a first side, and the image to be identified has a first edge, and based on the corresponding relationship and the multiple text contents, the multiple text categories are determined, including: when the corresponding relationship indicates that the second box to be processed is not located in the area covered by the at least one title box in the image to be identified, in response to the distance between the first edge of the second box to be processed and the first edge of the image to be identified being less than a preset distance and the text content corresponding to the second box to be processed contains title feature information, determining that the text category of the second box to be processed is the title.

[0010] For example, in the image processing method provided by at least one embodiment of the present disclosure, based on the image to be identified and the multiple area information, the multiple contents to be typeset are typeset to obtain a typeset document corresponding to the image to be identified, including: based on the multiple area information and the image to be identified, determining multiple typeset information corresponding to the multiple contents to be typeset respectively; based on the multiple typeset information, typeset the multiple contents to be typeset to obtain the typeset document.

[0011] For example, in the image processing method provided in at least one embodiment of the present disclosure, based on the multiple area information and the image to be identified, multiple layout information corresponding to the multiple contents to be typeset are determined, including: classifying the image to be identified through a classification model to determine the image category of the image to be identified; acquiring a layout template corresponding to the image category according to the image category; and determining the multiple layout information according to the layout template and the multiple area information.

[0012] For example, in the image processing method provided by at least one embodiment of the present disclosure, the multiple typesetting information are determined according to the typesetting template and the multiple area information, including: for the i-th content to be typeset among the multiple contents to be typeset: in response to the i-th content to be typeset being text content, determining the area information of the area box corresponding to the i-th content to be typeset, and determining the text category of the i-th content to be typeset according to the area information; determining the typesetting information corresponding to the i-th content to be typeset according to the typesetting template and the text category of the i-th content to be typeset, wherein i is a positive integer and is less than or equal to the total number of the multiple contents to be typeset.

[0013] For example, in the image processing method provided by at least one embodiment of the present disclosure, the multiple contents to be typeset are typeset based on the multiple typesetting information to obtain the typeset document, including: processing the multiple contents to be typeset to obtain multiple display contents; determining the positional relationship between the multiple display contents; and typeset processing the multiple display contents based on the positional relationship between the multiple display contents and the multiple typesetting information to obtain the typeset document.

[0014] For example, in the image processing method provided by at least one embodiment of the present disclosure, the multiple contents to be typeset include at least one first content to be typeset and at least one second content to be typeset, the at least one title box includes the at least one first content to be typeset, and the multiple contents to be typeset are processed to obtain multiple display contents, including: performing title number detection processing on the at least one first content to be typeset to obtain at least one intermediate display content; performing format processing on the at least one second content to be typeset and the at least one intermediate display content to obtain the multiple display contents.

[0015] For example, in the image processing method provided by at least one embodiment of the present disclosure, the at least one first content to be typeset is subjected to title number detection processing to obtain at least one intermediate display content, including: extracting the title number information corresponding to the at least one title frame to obtain at least one title number information; determining the positional relationship between the at least one title frame; judging whether there is a missing title number based on the positional relationship between the at least one title frame and the at least one title number information; in response to the existence of a missing title number: extracting the missing title number information, determining the missing area corresponding to the missing title number information in the image to be identified, completing the missing title number information based on the missing area to obtain the missing display content corresponding to the missing area, taking the missing display content and the at least one first content to be typeset as the at least one intermediate display content; in response to the absence of a missing title number: taking the at least one first content to be typeset as the at least one intermediate display content.

[0016] For example, in the image processing method provided by at least one embodiment of the present disclosure, determining the positional relationship between the multiple display contents includes: determining the positions of the multiple area boxes in the image to be identified based on the position information in the multiple area information; determining the positional relationship between the multiple contents to be typeset based on the positions of the multiple area boxes in the image to be identified; determining the positional relationship between the multiple display contents based on the positional relationship between the multiple contents to be typeset.

[0017] For example, in the image processing method provided by at least one embodiment of the present disclosure, the positional relationship between the multiple display contents is determined according to the positional relationship between the multiple contents to be typeset, including: determining whether the image to be identified includes multiple image partitions according to the positions of the multiple area boxes in the image to be identified; in response to the image to be identified including multiple image partitions, determining multiple sets of contents to be typeset corresponding to the multiple image partitions respectively, and determining the positional relationship between the multiple image partitions in the image to be identified; based on the positional relationship between the multiple image partitions, determining the positional relationship between the multiple sets of contents to be typeset; and determining the positional relationship between the multiple display contents based on the positional relationship between the multiple sets of contents to be typeset and the positional relationship between the multiple contents to be typeset.

[0018] For example, in the image processing method provided by at least one embodiment of the present disclosure, based on the positional relationship between the multiple display contents and the multiple layout information, the multiple display contents are layout processed to obtain the layout document, including: based on the multiple layout information, the multiple display contents are layout processed to obtain multiple layout display contents; the multiple layout display contents are arranged in sequence according to the positional relationship between the multiple display contents to obtain the layout document.

[0019] For example, in the image processing method provided by at least one embodiment of the present disclosure, the text category includes handwritten text, and in response to the text category of the first text content among the multiple text contents being handwritten text, and the typeset document does not contain the first text content, the image to be identified is identified to obtain multiple area boxes, multiple area information corresponding one-to-one to the multiple area boxes, and multiple contents to be typeset, and also includes: deleting the first text content from the multiple text contents to obtain the remaining at least one text content, wherein the multiple contents to be typeset include the remaining at least one text content but do not include the first text content.

[0020] At least one embodiment of the present disclosure provides an image processing device, comprising: an acquisition unit, configured to acquire an image to be recognized; an identification unit, configured to recognize the image to be recognized, so as to obtain a plurality of area frames, a plurality of area information corresponding one by one to the plurality of area frames, and a plurality of contents to be typeset; a typesetting unit, configured to typeset the plurality of contents to be typeset based on the image to be recognized and the plurality of area information, so as to obtain a typeset document corresponding to the image to be recognized.

[0021] At least one embodiment of the present disclosure provides an electronic device, comprising: a memory, which non-transiently stores computer-executable instructions; and a processor, configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the image processing method according to any embodiment of the present disclosure.

[0022] At least one embodiment of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the image processing method according to any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0024] Figure 1 A schematic flowchart of an image processing method provided by at least one embodiment of the present disclosure;

[0025] Figure 2 A schematic diagram of an image to be recognized provided in at least one embodiment of the present disclosure;

[0026] Figure 3 A schematic diagram of an image to be identified with a region frame provided in at least one embodiment of the present disclosure;

[0027] Figure 4A for Figure 1 An example flow chart of step S30 in the image processing method shown;

[0028] Figure 4B for Figure 4A An example flow chart of step S302 in the image processing method shown;

[0029] Figure 4C A schematic diagram of a typeset document containing article paragraphs provided in an embodiment of the present disclosure;

[0030] Figure 4D An embodiment of the present disclosure provides Figure 2 A schematic diagram of a typeset document corresponding to the image to be recognized shown;

[0031] Figure 5 A schematic block diagram of an image processing device provided by at least one embodiment of the present disclosure;

[0032] Figure 6 A schematic diagram of an electronic device provided for at least one embodiment of the present disclosure;

[0033] Figure 7 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0035] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits the detailed description of some known functions and known components.

[0036] At least one embodiment of the present disclosure provides an image processing method, an image processing device, an electronic device and a non-transitory computer-readable storage medium. The image processing method includes: acquiring an image to be identified; identifying the image to be identified to obtain a plurality of area frames, a plurality of area information corresponding to the plurality of area frames, and a plurality of contents to be typeset; and typeset the plurality of contents to be typeset based on the image to be identified and the plurality of area information to obtain a typeset document corresponding to the image to be identified.

[0037] At least one embodiment of the present disclosure provides an image processing method that processes an image to be identified through a pre-trained model to obtain a plurality of contents to be typeset and their category information, thereby obtaining a corresponding typesetting format according to the category information of the contents to be typeset to typeset the contents to be typeset, and finally obtaining a typeset document. The image processing method can obtain an electronic document corresponding to the image to be identified, thereby facilitating various operations on the image to be identified, such as storage, transmission, management, backup, printing, and the like.

[0038] The image processing method provided in the embodiment of the present disclosure can be applied to the image processing device provided in the embodiment of the present disclosure, and the image processing device can be configured on an electronic device. The electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a hardware device such as a mobile phone or a tablet computer.

[0039] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

[0040] Figure 1 A schematic flowchart of an image processing method provided for at least one embodiment of the present disclosure.

[0041] like Figure 1 As shown, the image processing method provided by at least one embodiment of the present disclosure includes steps S10 to S30.

[0042] Step S10: Acquire the image to be recognized.

[0043] Step S20: recognizing the image to be recognized to obtain a plurality of area frames, a plurality of area information corresponding to the plurality of area frames, and a plurality of contents to be typeset.

[0044] Step S30: Based on the image to be identified and the multiple area information, typeset the multiple contents to be typeset to obtain a typeset document corresponding to the image to be identified.

[0045] In some embodiments of the present disclosure, the image to be identified in step S10 is an image containing a question, for example, the image to be identified is an image of a test paper, homework, exercise book, etc. The test paper, homework, exercise book, etc. can be a paper file, so that the electronic document of the paper test paper can be obtained by the image processing method provided by the embodiment of the present disclosure, so that the test paper, etc. can be stored, transmitted, managed, backed up, printed, etc. The test paper can be a test paper of various subjects, such as Chinese, mathematics, foreign languages ​​(such as English, etc.), and similarly, the exercise book can also be an exercise book of various subjects, etc.

[0046] For example, the topic may include text content, for example, the text content may include text in various languages, for example, Chinese (Chinese characters and / or pinyin), English, Japanese, etc. For example, the text content may also include various numbers (Chinese numbers, Roman numbers, Arabic numbers, etc.), symbols (for example, greater than symbol, less than symbol, percent sign, etc.), graphics (circle, rectangle, etc.), for example, the text content may have various fonts, various colors, etc. For example, the text content may include printed text content and handwritten text content, and the handwritten text content may include, for example, handwritten words and letters, handwritten numbers, handwritten symbols and graphics, etc.

[0047] For example, the title may also include other types of information such as pictures or tables. The present disclosure does not specifically limit the content included in the title in the image to be recognized.

[0048] For example, the image to be identified may be an image captured by an image acquisition device (e.g., a digital camera or a mobile phone, etc.), and the image to be identified may be a grayscale image or a color image. It should be noted that the image to be identified refers to a form in which the object to be processed (e.g., a test paper, homework, exercise book, etc.) is presented in a visual manner, such as a picture of the object to be processed. For another example, the image to be identified may also be obtained by scanning or the like. For example, the image to be identified may be an image directly acquired by an image acquisition device, or may be an image obtained after preprocessing the acquired image.

[0049] For example, the typeset document may be an electronic document.

[0050] Figure 2 A schematic diagram of an image to be recognized provided by at least one embodiment of the present disclosure. Figure 2 As shown, the image to be identified is a test paper image, and the image to be identified contains multiple questions. Figure 2 The multiple areas divided by the black boxes in the example correspond to multiple questions. For example, each question here refers to a small question ( Figure 2 In the example, “1. Fill in the appropriate unit in the brackets”, etc.), multiple questions form a big question (such as “1. Fill in the blanks”, “2. Multiple choice”, etc.). Figure 2 As shown in the title 1 marked in the figure, title 1 may include text, and the text may include symbols, words, numbers, handwritten text, etc.; for example, Figure 2 As shown in the title 2 marked in the figure, title 2 can include text and tables; for example, Figure 2 As shown in the question 3 marked in the figure, question 3 may include text and pictures. By applying the image processing method provided by at least one embodiment of the present disclosure to the test paper, a typesetting document corresponding to the test paper can be obtained.

[0051] It should be noted that the images to be identified and the topics contained in the images to be identified in the present disclosure are not limited to Figure 2 In the case shown, the image to be recognized may also be an exercise book, etc., and the question may also refer to other question forms. For example, each question included in the image to be recognized may also refer to a large question.

[0052] For example, a pre-trained object detection model can be used to determine multiple area boxes in the image to be identified and area information corresponding to the multiple area boxes. For example, the area information may include position information of the area box in the image to be identified; then, the text content corresponding to the text box in the area box is determined by the text recognition model, and the text content is used as the content to be typeset. For another example, a pre-trained model can be used to directly obtain multiple area boxes in the image to be identified, area information corresponding to the multiple area boxes, and text content corresponding to the text box in the area box, and the text content is used as the content to be typeset. Here, the pre-trained model can complete the functions of the aforementioned object detection model and text recognition model. In other words, the recognition model for identifying the image to be identified can be multiple models or one model, and the present disclosure does not limit this.

[0053] For example, step S20 may include: identifying the image to be identified through an object detection model to obtain multiple area boxes and multiple area information, wherein the multiple area boxes include multiple first text boxes; identifying the multiple first text boxes through a text recognition model to obtain multiple text contents corresponding to the multiple first text boxes.

[0054] For example, the multiple contents to be typeset include one or more of the multiple text contents.

[0055] For example, step S20 may further include: determining a plurality of text categories corresponding to a plurality of first text boxes in a one-to-one manner according to a plurality of area information and a plurality of text contents.

[0056] For example, the area information corresponding to any first text box among the multiple first text boxes includes the text category of the any first text box.

[0057] For example, the object detection model can be a classification model of a pre-trained neural network, such as Faster R-CNN (Faster Region-Convolutional Neural Networks), R-FCN (Region-based Fully Convolutional Network) and other models. For example, based on the pre-trained object detection model, the text, picture and other regions in the image to be identified can be identified, and different classifications can be marked for different region boxes. For example, the region box can include a text box, a picture box, a title box, etc. For example, the title box can be composed of at least one text box and / or at least one picture box. For example, the title box can be Figure 2The box corresponding to "Topic 1" in the figure, that is, one topic corresponds to one topic box; for example, in some embodiments, the area box may also include a student information box, a title box, etc. Since student information, titles, etc. usually have specific formats and text features, the object detection model can also be trained so that the object detection model can directly process the image to be recognized to obtain the title box and the student information box.

[0058] It should be noted that the title box and the student information box can also both belong to text boxes, that is, the object detection model does not need to classify the area boxes corresponding to student information, titles, etc., but can directly classify the area boxes corresponding to student information, titles, etc. as text boxes.

[0059] For example, when the plurality of area frames further include at least one picture frame, step S20 may further include: extracting at least one picture to be laid out corresponding to the at least one picture frame. In this case, the plurality of contents to be laid out further include at least one picture to be laid out.

[0060] For example, in some embodiments, the image to be recognized contains complex mathematical forms or mathematical formulas, such as vertical equations in mathematical operations, etc. The mathematical forms or mathematical formulas can be used as frames, and the mathematical forms or mathematical formulas can be used as a part of the typesetting document in the form of pictures, thereby simplifying the generation of the typesetting document. For example, for the vertical equation, the text content of each line in the vertical equation can also be obtained, and the text content in the vertical equation can be typeset according to the positional relationship between the text content in the vertical equation and the template corresponding to the vertical equation to generate a typeset vertical equation, and the typeset vertical equation can be used as a part of the typesetting document.

[0061] For example, in some embodiments, a text box may appear in the image frame obtained by object detection model recognition, that is, some text content in the image frame will also have corresponding text boxes. Before obtaining the text content through the text recognition model, this part of the text box needs to be removed, and the image frame and the text content in the frame should be regarded as a whole frame.

[0062] For example, multiple area boxes also include at least one second text box, at least one second text box is located in the area where at least one frame is located, any frame in at least one frame includes a picture, and at least one picture to be typeset corresponding to at least one frame is extracted, including: in response to the area covered by any frame in the image to be identified including the area covered by N second text boxes in the image to be identified, the text contents corresponding to the N second text boxes and the picture in any frame are taken as a whole as the picture to be typeset corresponding to any frame, where N is a positive integer.

[0063] For example, when the multiple area boxes also include a student information box, step S20 may further include: identifying the student information box through a text recognition model to obtain student information, wherein the multiple contents to be typeset also include student information.

[0064] For example, when the multiple area boxes also include a title box, step S20 may further include: recognizing the title box through a text recognition model to obtain title information, wherein the multiple contents to be typeset also include title information.

[0065] For example, some images to be identified may have special information in the header and footer that needs to be displayed in the layout document. The multiple area frames obtained by the object detection model may also include the header frame and the footer frame. Step S20 may also include: identifying the header frame and the footer frame by the text recognition model to obtain the header information and the footer information, wherein the multiple contents to be typeset also include the header information and the footer information. Then in step S30, the header information and the footer information are typeset based on the pre-set header and footer format to generate a layout document with the header information and the footer information.

[0066] It should be noted that student information, title information, header information, footer information, etc. can all be expressed in text.

[0067] For example, there may be a table in the image to be identified, and the table can be identified as a table frame according to the table recognition model and converted into an electronic table. For example, the multiple area frames also include at least one table frame, and the area information corresponding to each table frame includes table information. Step S20 may also include: identifying the image to be identified by the table recognition model to obtain at least one table frame; based on the at least one table frame and the table information corresponding to the at least one table frame, generating at least one table content corresponding to the at least one table frame, wherein the multiple contents to be typeset also include at least one table content.

[0068] For example, in some embodiments, a table in the image to be identified corresponds to a table box, a table box may include multiple text boxes, each text box corresponds to the text content in the table, and the table information includes the positional relationship between the table box and the multiple text boxes included in the table box. For example, a table is generated based on the number of rows and columns of the table, and the text content in the text box is filled into the corresponding position in the generated table to generate a table content. It should be noted that other table recognition and generation methods can also be used to generate table content, and the present disclosure does not limit this.

[0069] For example, the text recognition model may include a character recognition model based on a neural network, such as an OCR (Optical Character Recognition) model, etc. The text recognition model can recognize the text content of a text box. For example, the text recognition model can use the recognized text content of the text box as the content to be typeset. For example, the text content here may include printed text content and handwritten text content. That is, the text recognition model can output printed text content and handwritten text content without distinction as the content to be typeset.

[0070] For example, in some embodiments, the text recognition model can also identify the type of text content, such as printed type or handwritten type. When the type of text content is handwritten type, a specific typesetting format different from that of printed type text content is set for it, such as setting the font of handwritten type text content to a handwritten font, etc., to generate a typeset document.

[0071] For example, in some embodiments, the text category includes handwritten text. When the type of the text content is a handwritten type, the text category of the text box is handwritten text. For example, in response to the text category of the first text content among multiple text contents being handwritten text, and the typeset document does not contain the first text content, step S20 may also include: deleting the first text content from the multiple text contents to obtain at least one remaining text content, wherein the multiple contents to be typeset include the remaining at least one text content but do not include the first text content.

[0072] It should be noted that “printed text content” does not only refer to text, characters, graphics and other content input on an electronic device through an input device, but also may refer to printed handwritten content of a user.

[0073] That is, by applying the image processing method provided by the present disclosure to the image to be identified of the test paper, exercise book, etc. with handwritten content, a document of the test paper, exercise book, etc. with the handwritten content removed can be generated, so that the test paper, exercise book, etc. can be easily repeated. It should be noted that the present disclosure can also identify the text category as handwritten text according to actual needs, and use it as part of the content to be typeset to generate a typeset document, and the present disclosure does not limit this.

[0074] In order to obtain a better display effect, it is necessary to set different typesetting formats for different texts, such as font, font size, bold, italic, indentation, etc. The text box can be further subdivided to determine the different text categories of different text boxes, so as to typeset the text content according to the format corresponding to the text category to obtain a typeset document with better typesetting effect.

[0075] For example, the image to be identified is an image containing at least one title, the multiple area boxes also include at least one title box corresponding to the at least one title, each title box includes at least one first text box in the area covered by the image to be identified, each area information includes position information of the area box corresponding to each area information in the image to be identified, and determining multiple text categories corresponding to the multiple first text boxes based on the multiple area information and the multiple text contents can include: determining the correspondence between at least one title box and the multiple first text boxes based on the position information in the multiple area information; and determining the multiple text categories based on the correspondence and the multiple text contents.

[0076] For example, multiple text categories include title, where "title" refers to Figure 2 The category names of the categorized questions such as "multiple choice questions" and "fill in the blank questions" shown in the figure need to be specially formatted (such as increasing the font size, making it bold, etc.), so it is necessary to identify such text boxes from multiple text boxes.

[0077] For example, at least one title box includes a first title box, in a first direction, the first title box has a first edge, and multiple first text boxes include a first to-be-processed box. Based on the correspondence and multiple text contents, multiple text categories are determined, which may include: in response to the correspondence indicating that the first to-be-processed box is located in the area covered by the first title box in the image to be identified, and there is no area box between the first to-be-processed box and the first edge, determining that the text category of the first to-be-processed box is the name of the main question; or, in response to the correspondence indicating that the first to-be-processed box is located outside the area covered by the first title box in the image to be identified and there is no area box between the first to-be-processed box and the first edge, and determining that the text content corresponding to the first to-be-processed box contains feature information of the main question, determining that the text category of the first to-be-processed box is the name of the main question.

[0078] It should be noted that “the first to-be-processed frame is located within the area covered by the first title frame in the image to be identified” means that on the image to be identified, the area covered by the first to-be-processed frame is located within the area covered by the first title frame. Figure 2 In the example shown, the first direction may be a vertical direction.

[0079] For example, when the first frame to be processed is located in the area covered by the first question frame in the image to be identified, since the name of the main question is usually located in the first row, the first side can be the upper side of the first question frame; for example, when the first frame to be processed is located outside the area covered by the first question frame in the image to be identified, since the name of the main question is usually adjacent to the upper side of the question frame, the first side can be the upper side of the first question frame. For example, the main question feature information indicates that the text content includes Chinese capital numbers, for example, the text content includes "one", "two", "three" and other contents.

[0080] For example, multiple text categories also include titles, where "title" refers to Figure 2 The text shown in the figure at the head of the test paper is: "Unit 1 Comprehensive Test Paper". Since it needs to be specially formatted (such as larger font size, bold display, etc.), it is necessary to identify this type of text box from multiple text boxes.

[0081] For example, multiple first text boxes include a second box to be processed, in a first direction, the second box to be processed has a first side, and the image to be identified has a first edge, and based on the correspondence and multiple text contents, multiple text categories are determined, including: when the correspondence indicates that the second box to be processed is not located in an area covered by at least one title box in the image to be identified, in response to the distance between the first side of the second box to be processed and the first edge of the image to be identified being less than a preset distance and the text content corresponding to the second box to be processed contains title feature information, determining that the text category of the second box to be processed is a title.

[0082] Since the title is usually located at the head of the test paper and close to the upper side of the test paper, the first edge can be the upper side of the image to be identified, and the first side can be the upper side of the second box to be processed. Therefore, it is possible to determine whether the text category of the second box to be processed is a title based on the position of the second box to be processed in the image to be identified and its corresponding text content.

[0083] Figure 3 A schematic diagram of an image to be identified with a region frame provided in at least one embodiment of the present disclosure. Figure 3 As shown, Figure 3 The multiple boxes in the figure are multiple area boxes obtained by identifying the image to be identified in step S20. For example, the multiple area boxes include image boxes and text boxes. For example, the text box located at the top of the image to be identified is a text box with a text category of title; for example, each title box corresponds to a major question, for example, title box 1 corresponds to the first major question (such as Figure 3 1. Calculate in vertical form) and question box 2 corresponds to the fifth question (such as Figure 3 For example, the text box in the first row of the title box is a text box with the text category of the title name (such as Figure 3 For example, a text box with a text content type of handwriting is a text box with a text category of handwriting text (such as Figure 3 Chinese text box “Answer: Xiaocong’s home is close to the school”).

[0084] It should be noted that Figure 3 The area box shown is only an example of an area box. When the object detection model is trained differently, area boxes of different forms may be generated. For example, a text box may contain multiple lines of text content, etc. This disclosure does not limit this.

[0085] For example, after obtaining the content to be typeset, the format of the content to be typeset may be adjusted according to different typesetting formats corresponding to different content to be typeset, so as to generate a typeset document.

[0086] Figure 4A for Figure 1 FIG. 1 is an example flow chart of step S30 in the image processing method shown in FIG. Figure 4A As shown, step S30 in the image processing method may specifically include steps S301 - S302 .

[0087] In step S301, based on a plurality of area information and an image to be identified, a plurality of layout information corresponding to a plurality of contents to be layout are determined.

[0088] For example, step S301 may include: classifying the image to be identified through a classification model to determine the image category of the image to be identified; obtaining a layout template corresponding to the image category according to the image category; and determining multiple layout information according to the layout template and the multiple area information.

[0089] For example, since the text characteristics of different subjects are different, a classification model can be used to divide the image to be identified according to subjects, such as Chinese, mathematics, English, etc., to obtain the typesetting template corresponding to the subject category. Of course, other classification methods can also be used as needed, and the present disclosure does not limit this.

[0090] For example, a typesetting template can specify information such as the number of words per line, font size, font category, character spacing, line spacing, paragraph spacing, etc. For example, the Chinese font category uses Songti, and the English and number font category uses "Times New Roman". The title font size is larger (for example, size 3, etc.) and is dark and bold, and the main title font size is larger (for example, size 3, etc.) and is dark and bold, etc.

[0091] For example, in step S301, multiple typesetting information are determined based on a typesetting template and multiple area information, which may include: for the i-th content to be typeset among multiple contents to be typeset: in response to the i-th content to be typeset being text content, determining the area information of the area box corresponding to the i-th content to be typeset, and determining the text category of the i-th content to be typeset based on the area information; determining the typesetting information corresponding to the i-th content to be typeset based on the typesetting template and the text category of the i-th content to be typeset, wherein i is a positive integer and is less than or equal to the total number of the multiple contents to be typeset.

[0092] The typesetting information specifies the typesetting format of the content to be typeset. For example, if the text category of the content to be typeset is the title of a major question, the typesetting information may include general typesetting formats such as the number of characters per line and character spacing, and may also include typesetting formats specially set for the title of a major question such as font size and font bolding.

[0093] By obtaining the typesetting information from a preset typesetting template to typeset the content to be typeset, the complexity of typesetting is reduced and a typesetting document with better typeset can be obtained.

[0094] In step S302, based on a plurality of typeset information, a plurality of contents to be typeset are typeset to obtain a typeset document.

[0095] For example, Figure 4B for Figure 4A FIG. 1 is an example flow chart of step S302 in the image processing method shown in FIG. Figure 4B As shown, step S302 in the image processing method may specifically include steps S3021-S3023.

[0096] In step S3021, a plurality of contents to be typeset are processed to obtain a plurality of display contents.

[0097] In step S3022, the positional relationship between the plurality of display contents is determined.

[0098] In step S3023, based on the positional relationship between the multiple display contents and the multiple layout information, the multiple display contents are layout processed to obtain a layout document.

[0099] For example, multiple contents to be typeset include at least one first content to be typeset and at least one second content to be typeset, and at least one title box includes at least one first content to be typeset. Step S3021 may include: performing question number detection processing on at least one first content to be typeset to obtain at least one intermediate display content; performing format processing on at least one second content to be typeset and at least one intermediate display content to obtain multiple display contents.

[0100] For example, the first content to be typeset here refers to the content to be typeset contained in the title box, and the second content to be typeset refers to other content to be typeset except the content to be typeset contained in the title box, such as title, student information, etc.

[0101] For example, in order to avoid missing questions due to incomplete images to be recognized, the first content to be typeset contained in the question box can be processed for question number detection to determine whether there are missing question numbers, and the missing questions can be retrieved to obtain a complete typeset document. Here, the question number can refer to the question number in the name of the big question, such as the capital numbers "one", "two", "three", etc., or it can refer to the question number of each small question, such as "1", "2", "3", etc.

[0102] For example, performing title number detection processing on at least one first content to be typeset to obtain at least one intermediate display content may include: extracting title number information corresponding to at least one title frame to obtain at least one title number information; determining the positional relationship between at least one title frame; judging whether there is a missing title number based on the positional relationship between at least one title frame and at least one title number information; in response to the existence of a missing title number: extracting the missing title number information, determining the missing area corresponding to the missing title number information in the image to be identified, completing the missing title number information based on the missing area to obtain the missing display content corresponding to the missing area, using the missing display content and at least one first content to be typeset as at least one intermediate display content; in response to the absence of a missing title number: using at least one first content to be typeset as at least one intermediate display content.

[0103] For example, judging whether there is a missing question number based on the positional relationship between at least one question frame and at least one question number information may include: sorting at least one question number information based on the positional relationship between at least one question frame to obtain a question number information sequence; determining that there is no missing question number in response to any two adjacent question number information in the question number information sequence being continuous; and determining that there is a missing question number in response to at least two adjacent question number information in the question number information sequence being discontinuous.

[0104] For example, completing the missing question number information based on the missing area to obtain the missing display content corresponding to the missing area may include: in response to the existence of a title box in the missing area, completing the missing question number information for the title box to obtain the missing display content corresponding to the missing area, wherein the missing display content corresponding to the missing area includes the missing question number information and the text content in the title box; in response to the absence of a title box in the missing area, identifying the missing area to obtain the missing display content corresponding to the missing area, wherein the missing display content corresponding to the missing area includes the text content in the missing area and the missing question number information.

[0105] For example, in some embodiments, there are skipped numbers in the identified question number information sequence, for example, the identified question number sequence is "1, 3, 4, 5...", then it is determined that the question with question number "2" is missing, that is, the missing question number information is "2", so the area between the question box with question number "1" and the question box with question number "3" is determined to be the missing area; then, it is detected whether there is a question box in the missing area, if there is a question box, the question number information can be completed by the text box that is not included in the question box in the missing area, or the question number information can be completed for the title box directly based on the obtained missing question number information; if there is no question box, step S20 can be executed again to identify the missing area to obtain the missing display content corresponding to the missing area.

[0106] After performing title number detection processing on at least one first content to be typeset, format processing can be performed on at least one second content to be typeset and at least one intermediate display content (hereinafter, at least one second content to be typeset and at least one intermediate display content are collectively referred to as content to be processed) to obtain multiple display contents.

[0107] For example, since the format in the typeset document may be different from the typeset in the image to be recognized, for example, the number of words in each line is different, and the title may contain multiple lines of text, it is necessary to append and segment the text content in different text lines in the same title, that is, the format processing may include text segmentation processing on the content to be processed to obtain multiple display contents, for example, dividing the contents to be typeset belonging to the same paragraph into one display content, that is, one display content may correspond to one paragraph. Each paragraph includes at least one line of text content.

[0108] For example, if two text boxes in the title box are adjacent to each other vertically, and the number of characters in the two text boxes is at least greater than a preset word count threshold, it is determined that the text contents in the two text boxes belong to the same paragraph.

[0109] For example, since the stem content of a question is usually a sentence, if the text content in a text box ends with a period, question mark or other symbol, and there is no other text content after the symbol in the horizontal direction, it can be determined that a paragraph ends.

[0110] For example, the need for segmentation can be determined based on the length of the text content. For example, if the first and third lines of text content in three consecutive lines are both relatively long, while the second line of text content is relatively short, and there is no image between the second and third lines of text content, then it is determined that the paragraph ends at the second line of text content, and the third line of text content belongs to a new paragraph.

[0111] For example, segmentation can be performed based on question type characteristics. For example, for a multiple-choice question, if the text content is an option and the option is located in a different line from the upper adjacent text box and the lower adjacent text box, then the option is judged to belong to a new independent paragraph.

[0112] For example, for a question containing multiple questions, if a sub-question number appears in the text content and the sub-question number is not located on the horizontal right side of the text content in the upper adjacent text box, then the text content containing the sub-question number is judged to belong to a new paragraph.

[0113] It should be noted that, in the above description of the present disclosure, the example of the text in the image to be recognized being typeset in horizontal rows from top to bottom is used for explanation, however, the present disclosure is not limited to this. If the text in the image to be recognized is typeset in vertical rows from right to left, then the above-mentioned "upper side" may be the "right side", the above-mentioned "lower side" may be the "left side", and the above-mentioned "horizontal right side" may be the "vertical upper side".

[0114] For example, since the first line of text in an article paragraph is usually indented by N characters, the article paragraph can be identified based on whether the first line of text in the content to be processed is indented by N characters, so as to be typeset according to the article format. For example, format processing may include segmenting the content to be processed in which the first line of text is indented by N characters, and typeset the first line of text according to a preset indentation format to obtain multiple display contents, where N is a positive integer greater than 1. It should be noted that if the article paragraph includes multiple paragraphs, the first line of text in each paragraph of the article paragraph is typeset according to the preset indentation format.

[0115] Figure 4C A schematic diagram of a layout document containing article paragraphs provided for at least one embodiment of the present disclosure. Figure 4C As shown, the title of the question is bold and black; the first line under the title of the question ( Figure 4C The question stem 1) shown in the figure is not indented by more than two characters, so it is typeset according to the typesetting information of the question stem; the first line of the text content of the English paragraph is indented by more than two characters, so it is judged that the English paragraph is an article paragraph, so the English paragraph is taken as a display content and typeset according to the preset indentation format; the text content after the English paragraph ( Figure 4C If the sub-question number appears in the question stem 2) shown in the figure, and the sub-question number is not located on the horizontal right side of the previous text line, then it is judged that each text content containing the sub-question number belongs to a paragraph, and the text content containing the sub-question number is used as a display content and typeset according to the typesetting information of the question stem.

[0116] For example, since the text recognition model may use special representation methods for special formats such as fractions, upper marks and lower marks, the format processing may also include format conversion of the content to be processed containing the special format to obtain the display content corresponding to the content to be processed containing the special format. For example, the text recognition model cannot intuitively represent the fraction. For example, the text recognition model will " " is expressed as "~1~2~", so it needs to be formatted to convert "~1~2~" into " " and is displayed as display content in the typeset document.

[0117] For example, special formats such as fractions, superscripts and subscripts can also be represented by dedicated formats, such as using LaTex representation methods to represent mathematical symbols, so that the text recognition model can directly output the display content in the typeset document without the need for formatting processing.

[0118] For example, step S3022 may include: determining the positions of multiple area boxes in the image to be identified based on the position information in the multiple area information; determining the positional relationship between multiple contents to be typeset based on the positions of the multiple area boxes in the image to be identified; determining the positional relationship between multiple display contents based on the positional relationship between the multiple contents to be typeset.

[0119] For example, Figure 2 As shown, the image to be identified may be divided into columns or pages, and each column or page is called an image partition. For example, a test paper usually has two or three pages on one side, which forms two or three image partitions. The image to be identified with columns or pages needs to be formatted so that the questions in the same column or page are grouped into the same page of the typesetting document. For example, this can be done based on the position information in the area information corresponding to the area box.

[0120] For example, determining the positional relationship between multiple display contents based on the positional relationship between multiple contents to be typeset may include: determining whether the image to be identified includes multiple image partitions based on the positions of multiple area boxes in the image to be identified; in response to the image to be identified including multiple image partitions, determining multiple sets of content to be typeset corresponding to the multiple image partitions respectively, and determining the positional relationship between the multiple image partitions in the image to be identified; based on the positional relationship between the multiple image partitions, determining the positional relationship between the multiple sets of content to be typeset; and determining the positional relationship between the multiple display contents based on the positional relationship between the multiple sets of content to be typeset and the positional relationship between the multiple contents to be typeset.

[0121] For example, when determining whether the image to be identified includes multiple image partitions, it can be determined based on the positions of the multiple area boxes in the image to be identified (for example, the coordinates in the image to be identified). Figure 2 As shown, when the content of the image to be identified is clearly divided into two columns, the horizontal coordinate value of the upper left corner coordinate of the title box will have a large interval. Therefore, it is possible to determine whether there are multiple image partitions based on this feature, and take the title boxes whose horizontal coordinate value difference of the upper left corner coordinate meets the preset threshold as a set of content to be typeset, and arrange the display content in a set of content to be typeset on the same page in the typeset document.

[0122] For example, step S3023 may include: based on multiple layout information, typeset multiple display contents to obtain multiple typeset display contents; and arrange the multiple typeset display contents in sequence according to the positional relationship between the multiple display contents to obtain a typeset document.

[0123] After obtaining a plurality of display contents, typeset processing is performed on the plurality of display contents based on the typesetting information, including: typeset processing is performed on the plurality of display contents according to the typesetting format specified by the typesetting information, such as font, font size, line spacing, paragraph spacing, character spacing, etc., to obtain a plurality of typeset display contents; thereafter, the plurality of typeset display contents are arranged in sequence according to the positional relationship between the plurality of display contents, thereby obtaining a typeset document.

[0124] Figure 4D An embodiment of the present disclosure provides Figure 2 The schematic diagram of the typeset document corresponding to the image to be recognized is shown.

[0125] like Figure 4D As shown, Figure 2 The schematic diagram of the layout document corresponding to the image to be recognized shown in the figure includes three pages, namely page (1), page (2) and page (3), wherein page (1) and page (2) show Figure 2 The content of the left image partition in the image to be identified is shown on page (3). Figure 2 The content of the right image partition in the image to be identified is shown.

[0126] like Figure 4D As shown, different text categories have different formats. Different text contents such as the title, question name, and question stem are displayed in different formats. For example, the format corresponding to the title is centered, the Chinese font is "Songti", the English font is "Times New Roman", the font size is size 3 and bold; the format corresponding to the question name is left-aligned, the Chinese font is "Songti", the English font is "Times New Roman", the font size is size 4 and bold; the format corresponding to the question stem is left-aligned, the Chinese font is "Songti", the English font is "Times New Roman", and the font size is size 12.

[0127] like Figure 4D As shown, tables and pictures are displayed in the layout document in the form of pictures (the table of the third question is not shown). For tables, electronic tables can also be generated by applying the aforementioned table recognition model and other methods, which will not be repeated here.

[0128] In addition, you can see that the Figure 2 The handwritten content in Figure 4D The typeset document shown is for Figure 2The blank typeset document corresponding to the object to be identified is shown, that is, the user can generate a typeset document of the test paper without handwritten content as needed, so as to repeat practice, backup, etc.

[0129] The image processing method provided by the present disclosure can process the image to be recognized to obtain the typeset document corresponding to the image to be recognized, and optimize the unique characteristics of the image to be recognized containing questions (for example, images of test papers, exercise books, etc. obtained by taking pictures or scanning, etc.). The recognition accuracy of such images to be recognized is higher, and the restoration degree of the typeset document corresponding to such images to be recognized is higher, providing a method for efficient and convenient test paper management, test paper storage, and wrong question recording.

[0130] At least one embodiment of the present disclosure further provides an image processing device, Figure 5 A schematic block diagram of an image processing device provided for at least one embodiment of the present disclosure.

[0131] like Figure 5 As shown, the image processing device 500 may include: an acquisition unit 501 , a recognition unit 502 and a typesetting unit 503 .

[0132] For example, these modules can be implemented by hardware (e.g., circuit) modules, software modules, or any combination of the two. The following embodiments are the same and will not be described in detail. For example, these units can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a field programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities and corresponding computer instructions.

[0133] For example, the acquisition unit 501 is configured to acquire an image to be recognized.

[0134] For example, the recognition unit 502 is configured to recognize the image to be recognized to obtain a plurality of area frames, a plurality of area information corresponding to the plurality of area frames one by one, and a plurality of contents to be typeset.

[0135] For example, the typesetting unit 503 is configured to typeset the multiple contents to be typeset based on the image to be identified and the multiple area information to obtain a typeset document corresponding to the image to be identified.

[0136] For example, the acquisition unit 501, the identification unit 502, and the typesetting unit 503 may include codes and programs stored in a memory; the processor may execute the codes and programs to implement some or all of the functions of the acquisition unit 501, the identification unit 502, and the typesetting unit 503 as described above. For example, the acquisition unit 501, the identification unit 502, and the typesetting unit 503 may be dedicated hardware devices, used to implement some or all of the functions of the acquisition unit 501, the identification unit 502, and the typesetting unit 503 as described above. For example, the acquisition unit 501, the identification unit 502, and the typesetting unit 503 may be a circuit board or a combination of multiple circuit boards, used to implement the functions described above. In an embodiment of the present application, the circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processor; and (3) firmware stored in the memory that is executable by the processor.

[0137] It should be noted that the acquisition unit 501 can be used to implement Figure 1 In step S10 shown in FIG. 1 , the identification unit 502 may be used to implement Figure 1 In step S20, the typesetting unit 503 can be used to implement Figure 1 Step S30 is shown. Therefore, for the specific description of the functions that can be implemented by the acquisition unit 501, the recognition unit 502 and the typesetting unit 503, reference can be made to the relevant description of steps S10 to S30 in the above-mentioned embodiment of the image processing method, and the repeated parts will not be repeated. In addition, the image processing device 500 can achieve technical effects similar to those of the above-mentioned image processing method, which will not be repeated here.

[0138] It should be noted that in the embodiments of the present disclosure, the image processing device 500 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited and can be composed of analog devices according to circuit principles, or can be composed of digital chips, or in other applicable ways.

[0139] At least one embodiment of the present disclosure further provides an electronic device, Figure 6 A schematic diagram of an electronic device provided according to at least one embodiment of the present disclosure.

[0140] For example, Figure 6As shown, the electronic device includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604. The processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604. The processor 601, the communication interface 602, the memory 603 and other components can also communicate through a network connection. The present disclosure does not limit the type and function of the network. It should be noted that Figure 6 The components of the electronic device shown are merely exemplary and non-limiting. The electronic device may also have other components according to actual application requirements.

[0141] For example, the memory 603 is used to store computer-readable instructions non-transiently. When the processor 601 is used to execute the computer-readable instructions, the image processing method according to any of the above embodiments is implemented. The specific implementation of each step of the image processing method and the related explanation content can be referred to the embodiment of the above image processing method, which will not be repeated here.

[0142] For example, other implementations of the image processing method implemented by the processor 601 executing computer-readable instructions stored in the memory 603 are the same as the implementations mentioned in the aforementioned method embodiment part, and will not be repeated here.

[0143] For example, the communication bus 604 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industrial Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0144] For example, the communication interface 602 is used to implement communication between the electronic device and other devices.

[0145] For example, the processor 601 and the memory 603 may be arranged on a server side (or a cloud side).

[0146] For example, the processor 601 can control other components in the electronic device to perform the desired functions. The processor 601 can be a device with data processing capabilities and / or program execution capabilities, such as a central processing unit (CPU), a network processor (NP), a tensor processing unit (TPU), or a graphics processing unit (GPU); it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The central processing unit (CPU) can be an X86 or ARM architecture, etc.

[0147] For example, the memory 603 may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and the processor 601 may execute the computer-readable instructions to implement various functions of the electronic device. Various applications and various data may also be stored in the storage medium.

[0148] For example, in some embodiments, the electronic device may further include an image acquisition component. The image acquisition component is used to acquire images. The memory 603 is also used to store the acquired images.

[0149] For example, the image acquisition component may be a camera of a smartphone, a camera of a tablet computer, a camera of a personal computer, a lens of a digital camera, or even a webcam.

[0150] For example, the image to be identified may be an original image directly acquired by the image acquisition component, or may be an image obtained after preprocessing the original image. Preprocessing may eliminate irrelevant information or noise information in the original image so as to better process the acquired image. Preprocessing may include, for example, image augmentation (Data Augment), image scaling, gamma correction, image enhancement or noise reduction filtering on the original image.

[0151] For example, for a detailed description of the process of performing image processing by the electronic device, reference may be made to the relevant description in the embodiment of the image processing method, and repeated parts will not be repeated.

[0152] Figure 7 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Figure 7 As shown, the storage medium 700 may be a non-transitory computer-readable storage medium, and one or more computer-readable instructions 701 may be non-transitory stored on the storage medium 700. For example, when the computer-readable instructions 701 are executed by a processor, one or more steps in the image processing method described above may be executed.

[0153] For example, the storage medium 700 may be applied to the above-mentioned electronic device. For example, the storage medium 700 may include a memory in the electronic device.

[0154] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other applicable storage media.

[0155] For example, the description of the storage medium 700 may refer to the description of the memory in the embodiment of the electronic device, and the repeated parts will not be repeated.

[0156] There are a few points to note about this disclosure:

[0157] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure, and other structures may refer to the general design.

[0158] (2) For the sake of clarity, the thickness and size of layers or structures are exaggerated in the drawings used to describe the embodiments of the present invention. It is understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element, or there may be intervening elements.

[0159] (3) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0160] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. An image processing method, comprising: Obtain an image to be recognized; Recognize the image to be recognized to obtain a plurality of area frames, a plurality of area information corresponding to the plurality of area frames one by one, and a plurality of contents to be typeset; Based on the image to be identified and the plurality of area information, typeset the plurality of contents to be typeset to obtain a typeset document corresponding to the image to be identified; The plurality of area boxes include a plurality of first text boxes and at least one title box; the area information corresponding to each of the area boxes includes position information of the area box in the image to be identified; The image processing method further comprises: Determining a correspondence between the at least one title box and the plurality of first text boxes according to position information in the plurality of region information, wherein the correspondence is used to indicate whether the first text box is located within a region covered by the title box in the image to be recognized, and whether there is any region box between the first text box and a first side of the title box; Determine the text categories corresponding to the plurality of first text boxes respectively based on the corresponding relationship and the plurality of text contents corresponding to the plurality of first text boxes respectively; The multiple area boxes include multiple first text boxes and at least one title box; the area information corresponding to each of the area boxes includes position information of the area box in the image to be identified, including: Classifying the image to be identified by a classification model to determine the image category of the image to be identified, wherein the image category includes a subject category; Acquire a typesetting template corresponding to the image category according to the image category; Determine the plurality of layout information according to the layout template and the plurality of area information; Based on the multiple typeset information, the multiple contents to be typeset are typeset to obtain the typeset document.

2. The method according to claim 1, wherein: The image to be identified is identified to obtain a plurality of area frames, a plurality of area information corresponding to the plurality of area frames, and a plurality of contents to be typeset, including: Identify the image to be identified by using an object detection model to obtain the multiple area frames and the multiple area information, wherein the multiple area frames include multiple first text frames; Recognize the multiple first text boxes through a text recognition model to obtain the multiple text contents corresponding to the multiple first text boxes one by one; The multiple contents to be typeset include one or more of the multiple text contents.

3. The method according to claim 2, wherein: The area information corresponding to any first text box among the multiple first text boxes includes the text category of the any first text box.

4. The method according to claim 2, wherein: The plurality of area frames also include at least one picture frame, The image to be identified is identified to obtain a plurality of area frames, a plurality of area information corresponding to the plurality of area frames one by one, and a plurality of contents to be typeset, and further comprising: extracting at least one picture to be typeset corresponding to each of the at least one picture frame, Among them, the multiple contents to be typeset also include the at least one picture to be typeset.

5. The method according to claim 1, wherein: The at least one title frame includes a first title frame, wherein the first title frame has a first side in a first direction, The plurality of first text boxes include a first to-be-processed box, the plurality of text categories include a title, Determining the multiple text categories based on the corresponding relationship and the multiple text contents includes: In response to the corresponding relationship indicating that the first frame to be processed is located in the area covered by the first title frame in the image to be recognized, and there is no area frame between the first frame to be processed and the first edge, determining that the text category of the first frame to be processed is the title of the major question; or, In response to the correspondence indicating that the first frame to be processed is located outside the area covered by the first question frame in the image to be identified and there is no area frame between the first frame to be processed and the first edge, and it is determined that the text content corresponding to the first frame to be processed contains feature information of the main question, it is determined that the text category of the first frame to be processed is the name of the main question.

6. The method according to claim 1, wherein: The plurality of first text boxes include a second to-be-processed box, the plurality of text categories include titles, In a first direction, the second frame to be processed has a first side, and the image to be recognized has a first edge. Determining the multiple text categories based on the corresponding relationship and the multiple text contents includes: When the corresponding relationship indicates that the second box to be processed is not located in the area covered by the at least one title box in the image to be identified, in response to the distance between the first side of the second box to be processed and the first edge of the image to be identified being less than a preset distance and the text content corresponding to the second box to be processed contains title feature information, it is determined that the text category of the second box to be processed is the title.

7. The method according to claim 1, wherein: Based on the image to be identified and the plurality of area information, typeset the plurality of contents to be typeset to obtain a typeset document corresponding to the image to be identified, including: Based on the plurality of area information and the to-be-recognized image, determining a plurality of layout information corresponding to the plurality of to-be-layout contents respectively; Based on the multiple typeset information, the multiple contents to be typeset are typeset to obtain the typeset document.

8. The method according to claim 1, wherein: Determining the plurality of layout information according to the layout template and the plurality of area information includes: For the i-th content to be typeset among the multiple contents to be typeset: In response to the i-th content to be typeset being text content, determining area information of an area box corresponding to the i-th content to be typeset, and determining a text category of the i-th content to be typeset according to the area information; Determine the typesetting information corresponding to the i-th content to be typeset according to the typesetting template and the text category of the i-th content to be typeset, Wherein, i is a positive integer and is less than or equal to the total number of the plurality of contents to be typeset.

9. The method according to claim 7, wherein: Based on the plurality of typeset information, typeset the plurality of contents to be typeset to obtain the typeset document, including: Processing the plurality of contents to be typeset to obtain a plurality of display contents; Determining a positional relationship between the plurality of display contents; Based on the positional relationship between the multiple display contents and the multiple layout information, the multiple display contents are layout processed to obtain the layout document.

10. The method according to claim 9, wherein: The plurality of contents to be typeset include at least one first content to be typeset and at least one second content to be typeset, and the at least one title box includes the at least one first content to be typeset. The plurality of contents to be typeset are processed to obtain a plurality of display contents, including: Performing a title number detection process on the at least one first content to be typeset to obtain at least one intermediate display content; The at least one second content to be typeset and the at least one intermediate display content are formatted to obtain the plurality of display contents.

11. The method according to claim 10, wherein: The at least one first content to be typeset is subjected to a title number detection process to obtain at least one intermediate display content, including: Extracting the question number information corresponding to the at least one question frame to obtain at least one question number information; Determining a positional relationship between the at least one title frame; Based on the positional relationship between the at least one question frame and the at least one question number information, it is determined whether there is a missing question number. In response to missing question numbers: Extract the missing question number information, determine the missing area corresponding to the missing question number information in the image to be identified, and complete the missing question number information based on the missing area to obtain the missing display content corresponding to the missing area. The omitted display content and the at least one first content to be typeset are used as the at least one intermediate display content, In response to the situation where there are no missing question numbers: The at least one first content to be typeset is used as the at least one intermediate display content.

12. The method according to claim 9, wherein: Determining the positional relationship between the plurality of display contents includes: Determining the positions of the plurality of area frames in the image to be identified according to the position information in the plurality of area information; Determining the positional relationship between the plurality of contents to be typeset based on the positions of the plurality of area frames in the image to be identified; The positional relationship between the multiple displayed contents is determined according to the positional relationship between the multiple contents to be typeset.

13. The method according to claim 12, wherein: Determining the positional relationship between the plurality of displayed contents according to the positional relationship between the plurality of contents to be typeset includes: determining, according to the positions of the plurality of area frames in the image to be identified, whether the image to be identified includes a plurality of image partitions; In response to the image to be identified including a plurality of image partitions, a plurality of sets of content to be typeset corresponding to the plurality of image partitions are determined, and a positional relationship between the plurality of image partitions in the image to be identified is determined. Determining the positional relationship between the plurality of to-be-layout content sets based on the positional relationship between the plurality of image partitions; Based on the positional relationship between the plurality of to-be-layout content sets and the positional relationship between the plurality of to-be-layout content sets, the positional relationship between the plurality of displayed content sets is determined.

14. The method according to claim 9, wherein: Based on the positional relationship between the plurality of display contents and the plurality of layout information, typeset processing is performed on the plurality of display contents to obtain the typeset document, including: Based on the plurality of layout information, the plurality of display contents are layout processed to obtain a plurality of layout display contents; The multiple typeset display contents are arranged in sequence according to the positional relationship between the multiple display contents to obtain the typeset document.

15. The method according to claim 3, wherein: The text category includes handwritten text, In response to the text category of a first text content among the plurality of text contents being handwritten text, and the typeset document not including the first text content, The image to be identified is identified to obtain a plurality of area frames, a plurality of area information corresponding to the plurality of area frames one by one, and a plurality of contents to be typeset, and further comprising: The first text content is deleted from the plurality of text contents to obtain at least one remaining text content, wherein the plurality of contents to be typeset include the at least one remaining text content but do not include the first text content.

16. An image processing device, comprising: An acquisition unit, configured to acquire an image to be recognized; an identification unit configured to identify the image to be identified to obtain a plurality of area frames, a plurality of area information corresponding to the plurality of area frames one by one, and a plurality of contents to be typeset; A typesetting unit, configured to typeset the plurality of contents to be typeset based on the image to be identified and the plurality of area information, so as to obtain a typeset document corresponding to the image to be identified; The plurality of area boxes include a plurality of first text boxes and at least one title box; the area information corresponding to each of the area boxes includes position information of the area box in the image to be identified; The image processing device further comprises: A determination unit is configured to determine a correspondence between the at least one title box and the plurality of first text boxes according to position information in the plurality of area information, wherein the correspondence is used to indicate whether the first text box is located in the area covered by the title box in the image to be identified, and whether there is any area box between the first text box and the first side of the title box; and determine text categories corresponding to the plurality of first text boxes respectively based on the correspondence and the plurality of text contents corresponding to the plurality of first text boxes respectively; wherein the typesetting unit is further configured to: Classifying the image to be identified by a classification model to determine the image category of the image to be identified, wherein the image category includes a subject category; Acquire a typesetting template corresponding to the image category according to the image category; The multiple typeset information are determined according to the typeset template and the multiple area information; and the multiple contents to be typeset are typeset based on the multiple typeset information to obtain the typeset document.

17. An electronic device comprising: A memory non-transitorily stores computer executable instructions; a processor configured to execute the computer executable instructions, Wherein, when the computer executable instructions are executed by the processor, the image processing method according to any one of claims 1-15 is implemented.

18. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the image processing method according to any one of claims 1-15 is implemented.

Images