Method, device and equipment for transferring picture to document and storage medium

Abstract

The invention discloses a method, device and equipment for transferring a picture to a document and a storage medium, the method comprises the steps that a first picture is input to a target detection model to obtain at least one object, the target detection model is used for detecting objects of different object types in the picture, and the object types comprise an image type, a formula type and a table type; for each object in the at least one object, constructing a first corresponding relationship between the object and an object identifier corresponding to the object to obtain at least one first corresponding relationship; processing the first picture based on the at least one object identifier corresponding to the at least one object to obtain a second picture; identifying the text information in the second picture to obtain a first document; and determining a second document corresponding to the first picture based on the first document and the at least one first corresponding relationship. According to the method, the objects of the table type and the image type can be identified and converted into the document, omission of the objects of the table type and the image type is avoided, and the conversion effect is improved.

Description

Technical Field

[0001] The present application belongs to the field of information processing technology, and in particular, relates to a method, device, equipment and storage medium for transferring images to documents. Background Art

[0002] With the rapid development of smart education and data-based learning, smart screens are playing an increasingly important role in classroom teaching and training. In order to facilitate students to review based on the content in class after class, the blackboard content written on the smart screen in class can be saved. In the prior art, when identifying the blackboard content in the smart screen, the text and formulas are mainly recognized, and the recognized text and formulas are converted into archived documents. Tables and images cannot be recognized. Therefore, when recognizing blackboard content with more complex forms such as text, formulas, tables and images, and converting them into documents, more content will be omitted in the document, making the effect of converting the blackboard content into a document poor. Summary of the invention

[0003] The embodiment of the present application provides an implementation scheme different from the prior art to solve the technical problem of poor conversion effect of converting complex blackboard content into documents.

[0004] In a first aspect, the present application provides a method for converting an image to a document, comprising: inputting a first image into a target detection model to obtain at least one object, wherein the target detection model is used to detect objects of different object types in the image, and the object types include image types, formula types, and table types; for each object in the at least one object, constructing a first correspondence between the object and an object identifier corresponding to the object to obtain at least one first correspondence; processing the first image based on at least one object identifier corresponding to the at least one object to obtain a second image; identifying text information in the second image to obtain a first document; and determining a second document corresponding to the first image based on the first document and the at least one first correspondence.

[0005] In a second aspect, the present application provides a device for converting an image to a document, comprising: an input unit, used to input a first image into a target detection model to obtain at least one object, wherein the target detection model is used to detect objects of different object types in the image, and the object types include image types, formula types and table types; a construction unit, used to construct, for each object in the at least one object, a first correspondence between the object and an object identifier corresponding to the object to obtain at least one first correspondence; a processing unit, used to process the first image based on at least one object identifier corresponding to the at least one object to obtain a second image; an identification unit, used to identify text information in the second image to obtain a first document; a determination unit, used to determine the second document corresponding to the first image based on the first document and the at least one first correspondence.

[0006] In a third aspect, the present application provides an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any method in the first aspect or any possible implementation manner of the first aspect by executing the executable instructions.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements any method in the first aspect or any possible implementation manner of the first aspect.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect or any possible implementation manner of the first aspect.

[0009] The present application provides a method for obtaining at least one object by inputting a first picture into a target detection model, wherein the target detection model is used to detect objects of different object types in the picture, and the object types include image type, formula type and table type; for each object in the at least one object, a first correspondence relationship between the object and the object identifier corresponding to the object is constructed to obtain at least one first correspondence relationship; the first picture is processed based on at least one object identifier corresponding to the at least one object to obtain a second picture; the text information in the second picture is recognized to obtain a first document; and a scheme for determining the second document corresponding to the first picture based on the first document and the at least one first correspondence relationship. The objects corresponding to the image type, formula type and table type detected by the target detection model are used. The first picture is processed using the object identifier corresponding to the object, and then the text information of the second picture containing the identifier information is recognized to obtain a first document. The identifier information in the first document is replaced based on the object corresponding to the identifier to obtain a second document. The objects containing table type, formula type and image type can all be recognized and converted into documents, avoiding the omission of objects of table type, graphic type and formula type, thereby improving the conversion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 A schematic diagram of a process of transferring an image to a document provided in one embodiment of the present application; Figure 2 A schematic diagram of the structure of a device for transferring images to documents provided in one embodiment of the present application; Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, but cannot be understood as limiting the present application.

[0012] The terms "first" and "second" etc. in the specification, claims and drawings of the embodiments of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0013] First, some terms in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0014] Markdown: A lightweight markup language designed to make writing and editing documents easier and more intuitive.

[0015] With the rapid development of smart education and data-based learning, smart large screens play an increasingly important role in classroom teaching and training. In order to facilitate students to review based on the content in the classroom after class, the blackboard content written on the smart large screen in the classroom can be saved. However, on the large screen, the content of handwritten blackboards is complex and the amount of information is large, and it is very difficult to convert the blackboard content into archived and editable documents. In the prior art, when identifying the blackboard content in the smart large screen, the text and formula are mainly identified, and the identified text and formula are converted into archived documents. Tables and images cannot be recognized. Therefore, when identifying blackboard content with more complex forms such as text, formulas, tables and images and converting them into documents, more content will be omitted in the document, making the effect of converting the blackboard content into documents poor. And the recognition effect of complex formulas is also poor, which easily leads to errors in the formulas after recognition, which in turn leads to poor results and low accuracy in converting the blackboard content containing complex formulas into documents.

[0016] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0017] Figure 1 A flowchart of a method for transferring images to documents is provided for an exemplary embodiment of the present application. The execution subject of the method may be a server. The method at least includes the following steps S101-S105: S101. Input a first image into a target detection model to obtain at least one object, wherein the target detection model is used to detect objects of different object types in the image, and the object types include image types, formula types, and table types.

[0018] Optionally, multiple sample images are obtained, and for each sample image in the multiple sample images, the object type of the object in the sample image is annotated, thereby obtaining multiple target sample images containing data annotations, wherein the object type includes image type, formula type, table type and text type. Based on the multiple target sample images, the detection model is classified and trained to obtain a target detection model that can be put into use. The objects detected by the target detection model include object content and object coordinate information.

[0019] Optionally, the image type may be represented as FIGURE, the formula type may be represented as MATH, the table type may be represented as TABEL, and the text type may be represented as TEXT.

[0020] Optionally, the object is a minimum rectangular frame image containing object content, and the minimum rectangular frame image is the smallest image in the first image that contains the object content. Specifically, when the target detection model detects the object content, the coordinate information of the minimum rectangular frame containing the object content is obtained, and the minimum rectangular frame image is obtained as the object based on the coordinate information.

[0021] Optionally, the first image is a first image among multiple first images obtained by segmenting a complete image. The complete image may be a handwritten blackboard image of a smart screen, a photographed blackboard or whiteboard image, a photographed document image, a book image or other image, etc., which will not be described in detail here.

[0022] Exemplarily, for ease of understanding, an example of a handwritten blackboard picture obtained on a smart large screen is used here for illustration, and the obtained handwritten blackboard picture is segmented to obtain multiple first pictures of appropriate lengths, and each of the multiple first pictures is processed as a first picture input to the target detection model. Among them, the appropriate length is a length suitable for input to the target detection model.

[0023] Optionally, segmenting the complete picture to obtain a plurality of first pictures includes the following steps S01-S05: S01, taking the first image edge of the complete image as a reference edge; S02, taking a line segment in the complete image that faces a preset direction, has a preset distance from the reference edge, and has the same length as the reference edge as a pending segmentation line; Optionally, the preset direction is a direction from the reference edge, perpendicular to the reference edge, and toward the inside of the complete image, wherein the reference edge may be the top edge of the complete image or the bottom edge of the complete image.

[0024] S03, determining whether the pending segmentation line meets a preset condition, if so, segmenting the complete image from the pending segmentation line, if not, determining a new pending segmentation line based on a preset moving interval, and segmenting the complete image from the new pending segmentation line; Optionally, in the aforementioned S03, the determining of a new pending dividing line based on a preset moving interval includes: taking the pending dividing line as the center, moving pixel by pixel to both sides of the pending dividing line respectively, and during the movement, if the number of pixel values ​​in the pending dividing line at a position that is a first preset value is greater than the first preset number, then the pending dividing line at the position is used as the new pending dividing line; wherein, the first preset value is the pixel value of the background area of ​​the complete picture, and when the number of pixel values ​​in the pending dividing line that is a second preset value is greater than the second preset number, the pending dividing line is deemed to meet the preset conditions.

[0025] Optionally, in the preset moving interval, the pending segmentation line whose distance from the reference edge is the preset distance is taken as the center, and the pixel is moved to both sides of the pending segmentation line respectively, and during the moving process, if at a certain position, the pixel values ​​of the pending segmentation line are all the first preset values, the pending segmentation line at the position is used as the new pending segmentation line; if during the moving process, the pixel values ​​of the pending segmentation line at any position are not all the first preset values, the pending segmentation line is moved to both sides of the pending segmentation line respectively, and pixel by pixel, and during the moving process, if at any position, the pixel values ​​of the pending segmentation line are not all the first preset values, When the number of pixel values ​​of the pending dividing line at a position that is a first preset value is greater than the first preset number, the pending dividing line at the position is used as a new pending dividing line; if the number of pixel values ​​of the pending dividing line at any position that is a first preset value is not greater than the first preset number, the pending dividing line at the target position is used as a new pending dividing line, wherein the target position is in the preset moving interval, and the number of pixel values ​​of the pending dividing line at the target position that is a first preset value is greater than the number of pixel values ​​of the pending dividing line at any position other than the target position in the preset moving interval that is the first preset value.

[0026] Exemplarily, if the background color is black, that is, the pixel value is (0, 0, 0), the preset distance is 2000 pixels, and the preset moving interval is to move the pending segmentation line upward by no more than 500 pixels and downward by no more than 500 pixels; taking the top edge of the complete image as the reference edge, if the pixel values ​​of the pixels of the pending segmentation line at a position 2000 pixels away from the reference edge are all (0, 0, 0), it means that there is no object at this position, which means that the complete image can be segmented at this position; if the pixel values ​​of the pixels of the pending segmentation line at a position 2000 pixels away from the reference edge are not all (0, 0, 0), it means that there is an object at this position, and the pending segmentation line needs to be moved one pixel upward or downward one pixel at a time, and the upward moving distance is not greater than 500 pixels, and the downward moving distance is not greater than 500 pixels. If the pixel values ​​of each pixel of the pending segmentation line at a certain position are all (0,0,0) during the moving process, the pending segmentation line at this position is used as the new pending segmentation line, indicating that the complete image can be segmented at this position; If no pending segmentation line with all pixel values ​​(0,0,0) is found in the preset position interval, the pending segmentation line with the highest percentage of pixel values ​​(0,0,0) in the preset position interval is used as the new pending segmentation line.

[0027] In summary, segmenting the complete image at a position that does not contain an object ensures the integrity of the first image after segmentation. If there is no position that does not contain an object, segmenting the complete image at a position that contains the highest background color ensures the integrity of the first image after segmentation to the greatest extent.

[0028] S04, determining whether the distance between the second picture edge of the complete picture in the preset direction and the dividing line closest to the second picture edge is less than the preset distance, if so, taking the pictures divided at each time as multiple first pictures, if not, taking the second picture edge of the complete picture in the preset direction and the dividing line closest to the second picture edge as new reference edges; S05. Return to the step of using the line segment in the complete picture that faces the preset direction, has a preset distance from the reference edge, and has the same length as the reference edge as the pending segmentation line, until the distance between the second picture edge of the complete picture in the preset direction and the segmentation line closest to the second picture edge is less than the preset distance.

[0029] S102: For each object in the at least one object, establish a first corresponding relationship between the object and an object identifier corresponding to the object to obtain at least one first corresponding relationship.

[0030] Optionally, when there is a formula object whose object type is a formula type in at least one of the objects, before constructing a first corresponding relationship between the formula object and the formula object, the method also includes: inputting the formula object into a preset formula recognition model to obtain formula information; constructing a first corresponding relationship between the formula object and the object identifier corresponding to the formula object, including: constructing a first corresponding relationship between the formula information and the object identifier corresponding to the formula object.

[0031] Optionally, when there is an image object whose object type is an image type among the at least one object, before constructing a first correspondence between the image object and the image object, the method also includes: storing the image object in a storage area to obtain address information for storing the image object; constructing a first correspondence between the image object and the object identifier corresponding to the image object, including: constructing a first correspondence between the address information and the object identifier corresponding to the image object.

[0032] Optionally, when there is a table object whose object type is a table type among the at least one object, before constructing a first correspondence between the table object and the table object, the method also includes: storing the table object in a storage area to obtain address information for storing the table object; constructing a first correspondence between the table object and the object identifier corresponding to the table object, including: constructing a first correspondence between the address information and the object identifier corresponding to the table object.

[0033] S103: Process the first picture based on at least one object identifier corresponding to the at least one object to obtain a second picture.

[0034] Optionally, the at least one object in the first picture is replaced with at least one object identifier corresponding to the at least one object to obtain a second picture. Specifically, for each object in the at least one object, the object in the first picture is replaced with the object identifier corresponding to the object until a first picture containing only text-type objects is obtained, and the first picture containing only text-type objects is used as the second picture.

[0035] Optionally, for each object in the at least one object, the original position area of ​​the object in the first picture is filled based on the background color of the first picture, and an object identifier corresponding to the object is inserted into the original position area, wherein the color of the object identifier is different from the background color of the first picture.

[0036] Exemplarily, when the object is detected as a first image object, the intercepted object is an object of the image type. The position area corresponding to the first image object in the first image is filled with the background color of the first image, and PIC_1 is inserted in the position area. The first image object is stored, and the first address information for storing the first image object is obtained, a first object relationship between the first address information and PIC_1 is constructed, and the first object relationship is stored in the image dictionary. When the object is detected as a second image object, the intercepted object is an object of the image type. The position area corresponding to the second image object in the first image is filled with the background color of the first image, and PIC_2 is inserted in the position area. The second image object is stored, and the second address information for storing the second image object is obtained, a first object relationship between the second address information and PIC_2 is constructed, and the first object relationship is stored in the image dictionary. By analogy, PIC_N can be used as the object identifier of the image type object, where N indicates that the corresponding object is the Nth image type object in the first image.

[0037] When the object is detected as the first table object, the intercepted object is an object of table type. The position area corresponding to the first table object in the first picture is filled with the background color of the first picture, and TABEL_1 is inserted in the position area. The first table object is stored, and the third address information for storing the first table object is obtained, a first object relationship between the third address information and TABEL_1 is constructed, and the first object relationship is stored in the table dictionary. When the object is detected as the second table object, the intercepted object is an object of table type. The position area corresponding to the second table object in the first picture is filled with the background color of the first picture, and TABEL_2 is inserted in the position area. The second table object is stored, and the fourth address information for storing the second table object is obtained, a first object relationship between the fourth address information and TABEL_2 is constructed, and the first object relationship is stored in the table dictionary. By analogy, TABEL_M can be used as the object identifier of an object of table type, where M indicates that the corresponding object is the Mth table type object in the first picture.

[0038] When the object is detected as a first formula object, the intercepted object is an object of the formula type. The position area corresponding to the first formula object in the first picture is filled with the background color of the first picture, and LATEX_1 is inserted in the position area. The first formula object is input into the formula recognition model for processing, the first formula information corresponding to the first formula object is obtained, the first object relationship between the first formula information and LATEX_1 is constructed, and the first object relationship is stored in the formula dictionary. When the object is detected as a second formula object, the intercepted object is an object of the formula type. The position area corresponding to the second formula object in the first picture is filled with the background color of the first picture, and LATEX_2 is inserted in the position area. The second formula object is input into the formula recognition model for processing, the second formula information corresponding to the second formula object is obtained, the first object relationship between the second formula information and LATEX_2 is constructed, and the first object relationship is stored in the formula dictionary. By analogy, LATEX_X can be used as the object identifier of the object of the formula type, where X indicates that the corresponding object is the Xth object of the formula type in the first picture.

[0039] The formula information is a mathematical expression, and the format of the first object relationship stored in the dictionary is a key-value pair format. In the formula dictionary, the object identifier is the key and the formula information is the value. In the table dictionary and the image dictionary, the object identifier is the key and the address information is the value.

[0040] S104: Identify the text information in the second image to obtain a first document.

[0041] Optionally, the second image is input into a visual language model to generate a first document including the at least one object identifier, wherein the format of the first document is Markdown. For relevant contents of the visual language model, please refer to the prior art, which will not be described in detail here.

[0042] Exemplarily, the visual language model may have preset requirements for identifying and processing the second image, and the preset requirements may be "identify the second image, maintain the layout and sequence, and organize it into a markdown format." The second image is input into the visual language model, and the visual language model uses the preset requirements to identify and process the second image to obtain a first document in markdown format containing at least one object identifier.

[0043] S105: Determine a second document corresponding to the first image based on the first document and the at least one first corresponding relationship.

[0044] Optionally, in the aforementioned S105, the second document corresponding to the first image is determined based on the first document and the at least one first correspondence, including: for each object identifier in at least one object identifier in the first document, using the object corresponding to the object identifier identical to the object identifier in the at least one first correspondence, to obtain the second document.

[0045] Specifically, when the at least one first corresponding relationship is stored in the form of a key-value pair, for each object identifier in at least one object identifier in the first document, the object identifier in the first document is replaced with the value corresponding to the key identical to the object identifier in the at least one first object relationship, thereby obtaining a second document. For the relevant explanation of the key-value pair, please refer to the prior art, which will not be described in detail here. The second document includes text information and formula information, and the text information includes address information of objects of the storage table type and address information of objects of the storage image type.

[0046] Exemplarily, the formula information stored in the formula dictionary replaces the object identifier corresponding to the formula information in the first document. The address information stored in the image dictionary replaces the object identifier corresponding to the address information in the first document. The address information stored in the table dictionary replaces the object identifier corresponding to the address information in the first document.

[0047] Optionally, the object type further includes a text type, each first image corresponds to a second document, and the method further includes the following steps S001-S004: S001, splicing multiple second documents corresponding to the multiple first images to obtain a first target document corresponding to the complete image; S002, obtaining at least one detection information of at least one object to be detected whose format type is a title format in the first target document, wherein the detection information includes the object type of the object to be detected, the address information of the object to be detected, and the number of text words of the object to be detected whose object type is the text type; S003, based on the at least one detection information, determining a target object that does not meet a preset title rule from the at least one object to be detected; Optionally, for each detection information in the at least one detection information, if the object type in the detection information is any of an image type, a table type or a formula type, the object to be detected corresponding to the detection information is deemed to be a target object that does not comply with the preset title rules. Or, if the object type in the detection information is a text type, and the number of text words in the detection information exceeds a preset number of words, the object to be detected corresponding to the detection information is deemed to be a target object that does not comply with the preset title rules. The preset number of words can be 6, 7, 8, etc., and the preset number of words can be set according to actual needs. The object type in the detection information is the object type of the object in the first picture corresponding to the information in the corresponding position area in the second document.

[0048] S004. Clear the title format of the target object in the first target document to obtain a second target document.

[0049] Optionally, the method further comprises the following steps S0111-S0115: S0111. Join the multiple second documents corresponding to the multiple first images to obtain a first target document corresponding to the complete image.

[0050] S0112. For the adjacent second document in the first target document, obtain the first coordinate information of the tail area of ​​the third document and the second coordinate information of the head area of ​​the fourth document; wherein the information in the tail area and the head area corresponds to the object type of the object in the first image being a text type; the third document is located above the fourth document in the first target document, and the third document is adjacent to the fourth document; wherein the first coordinate information includes the upper left vertex coordinates and the lower right vertex coordinates of the first line area. The second coordinate information includes the upper left vertex coordinates and the lower right vertex coordinates of the tail line area.

[0051] S0113. If the horizontal coordinate of the upper left vertex in the first coordinate information is not greater than the horizontal coordinate of the upper left vertex in the second coordinate information, clearing the title format of the text information corresponding to the second coordinate information in the first target document; or, S0114. If the absolute value of the vertical coordinate of the lower right vertex in the first coordinate information minus the vertical coordinate of the upper left vertex is not less than the absolute value of the vertical coordinate of the lower right vertex minus the vertical coordinate of the upper left vertex in the second coordinate information, then clear the title format of the text information corresponding to the second coordinate information in the first target document.

[0052] S0115. Stop executing until the fourth document is the second document at the bottom of the first target document.

[0053] Optionally, the title format may be preceded by a pound sign (#). Clearing the pound sign is equivalent to clearing the title format.

[0054] It should be understood that the second text recognized by the visual language model may miss object identifiers. To solve this problem, the method further includes the following steps S0001-S0002: S0001. When it is detected that a first number of object identifiers included in the first document is smaller than a second number of object identifiers included in the at least one first corresponding relationship, obtaining an object identifier to be supplemented that is missing from the first document relative to the at least one first corresponding relationship; S0002. Add the object corresponding to the to-be-supplemented object identifier into the second document.

[0055] Specifically, when the object identifier to be supplemented is an identifier corresponding to formula information, the second image is re-input into the visual language model, a new first document is regenerated using the visual language model, the new first document is used as the first document, and the second document corresponding to the first image is determined based on the first document and the at least one first correspondence. If the object identifier to be supplemented is address information of an object corresponding to an image type or a table type, the address information corresponding to the object identifier to be supplemented is supplemented in the tail area of ​​the second document.

[0056] Optionally, the method can be applied to intelligent blackboard recognition for large screens, and the book contents including text types, formula types, table types and image types can be converted into an archivable and editable Markdown data format.

[0057] In summary, the present application provides a method for obtaining at least one object by inputting a first image into a target detection model, wherein the target detection model is used to detect objects of different object types in the image, and the object types include image types, formula types, and table types; for each object in the at least one object, a first correspondence relationship between the object and the object identifier corresponding to the object is constructed to obtain at least one first correspondence relationship; the first image is processed based on at least one object identifier corresponding to the at least one object to obtain a second image; the text information in the second image is recognized to obtain a first document; and a scheme for determining the second document corresponding to the first image based on the first document and the at least one first correspondence relationship. The objects corresponding to the image type, formula type, and table type detected by the target detection model are used. The first image is processed using the object identifier corresponding to the object, and then the text information of the second image containing the identifier information is recognized to obtain a first document. The identifier information in the first document is then replaced based on the object corresponding to the identifier to obtain a second document. Objects containing table types, formula types, and image types can all be recognized and converted into documents, avoiding the omission of objects of table types, graphic types, and formula types, thereby improving the conversion effect.

[0058] Specifically, the visual language model is used to automatically identify the second image processed by the object identifier, obtain the first document in Markdown format, and then replace the object identifier in the first document with the value corresponding to the object identifier to obtain the second document in Markdown format that is easy to archive, edit, share and reuse. The first image is segmented to obtain an image size suitable for processing by the target detection model. The object type detection of the target detection model can quickly and accurately obtain objects of image type, table type and formula type, and the corresponding relationship between objects and object identifiers is constructed for different types of objects, so that the object in the image is replaced by the object identifier to obtain an image containing text information, thereby improving the accuracy and conversion efficiency of converting the image containing text information into storable text. The object identifier contained in the text can be used to obtain the corresponding object, and the conversion effect of the image, table and formula in the image is achieved, avoiding the problem of missing images, tables and formulas in the converted text due to poor recognition of images, tables and formulas, and avoiding the problem of incomplete or wrong images, tables and formulas in the converted text. The accuracy and conversion effect of the converted text are improved. By setting a unique object identifier for the object, it is ensured that the object corresponding to the object identifier can be accurately obtained.

[0059] In addition, according to this solution, a method for determining the title is also provided to avoid setting non-title information as the title, thereby ensuring that the title in the generated Markdown format text complies with the specification. This improves the readability of the second document and the user experience. This solution also provides a method for checking for missing object identifiers, and the values ​​corresponding to the missing object identifiers are supplemented in the second document, thereby ensuring that the second document contains all the information in the first image. This improves the completeness and accuracy of the document.

[0060] Figure 2 A schematic diagram of a device for transferring images to documents provided as an exemplary embodiment of the present application; wherein the device comprises: An input unit 21, used to input the first picture into the target detection model to obtain at least one object, wherein the target detection model is used to detect objects of different object types in the picture, and the object types include image types, formula types and table types; A construction unit 22, configured to construct, for each object in the at least one object, a first correspondence between the object and an object identifier corresponding to the object, to obtain at least one first correspondence; A processing unit 23, configured to process the first picture based on at least one object identifier corresponding to the at least one object to obtain a second picture; A recognition unit 24, configured to recognize text information in the second image to obtain a first document; The determination unit 25 is configured to determine a second document corresponding to the first image based on the first document and the at least one first corresponding relationship.

[0061] Optionally, in the device, the first picture is obtained by segmenting a complete picture to obtain one of multiple first pictures.

[0062] Optionally, the device is used to segment the complete image to obtain multiple first images, specifically for: taking the first image edge of the complete image as a reference edge; taking a line segment in the complete image that is oriented in a preset direction, is a preset distance away from the reference edge, and has the same length as the reference edge as a pending segmentation line; determining whether the pending segmentation line meets a preset condition, and if so, segmenting the complete image from the pending segmentation line; if not, determining a new pending segmentation line based on a preset moving interval, and segmenting the complete image from the new pending segmentation line; determining a second image edge of the complete image in the preset direction whether the distance between the edge of the second picture and the dividing line closest to the edge of the second picture is less than the preset distance, if so, take the pictures divided each time as multiple first pictures, if not, take the second picture edge of the complete picture in the preset direction and the dividing line closest to the edge of the second picture as new reference edges; return to execute the step of taking the line segment in the complete picture that faces the preset direction, has a preset distance from the reference edge, and has the same length as the reference edge as the pending dividing line, until the distance between the second picture edge of the complete picture in the preset direction and the dividing line closest to the edge of the second picture is less than the preset distance.

[0063] Optionally, the device is used to determine a new pending dividing line based on a preset moving interval, and is specifically used to: move pixel by pixel to both sides of the pending dividing line with the pending dividing line as the center, and during the movement, if the number of pixel values ​​in the pending dividing line at a position that is a first preset value is greater than the first preset number, then the pending dividing line at the position is used as a new pending dividing line; wherein, the first preset value is the pixel value of the background area of ​​the complete picture, and when the number of pixel values ​​in the pending dividing line that is a second preset value is greater than the second preset number, the pending dividing line is deemed to meet the preset conditions.

[0064] Optionally, in the device, the object type also includes a text type, and each first image corresponds to a second document. The device is also used to: splice the multiple second documents corresponding to the multiple first images to obtain the first target document corresponding to the complete image; obtain at least one detection information of at least one object to be detected whose format type is a title format in the first target document, wherein the detection information includes the object type of the object to be detected, the address information of the object to be detected, and the number of text words of the object to be detected whose object type is the text type; based on the at least one detection information, determine the target object that does not meet the preset title rules from the at least one object to be detected; clear the title format of the target object in the first target document to obtain the second target document.

[0065] Optionally, when the device is used, when there is a formula object of a formula type in at least one of the objects, before constructing a first corresponding relationship between the formula object and the formula object, the device is also used to: input the formula object into a preset formula recognition model to obtain formula information; constructing a first corresponding relationship between the formula object and the object identifier corresponding to the formula object includes: constructing a first corresponding relationship between the formula information and the object identifier corresponding to the formula object.

[0066] Optionally, the device is also used to: when it is detected that the first number of object identifiers contained in the first document is smaller than the second number of object identifiers contained in the at least one first correspondence, obtain the object identifier to be supplemented that is missing from the first document relative to the at least one first correspondence; and supplement the object corresponding to the object identifier to be supplemented in the second document.

[0067] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the device may perform the above method embodiment, and the above and other operations and / or functions of each module in the device are the corresponding processes in each method in the above method embodiment, respectively, and no further description is given here for the sake of brevity.

[0068] The above describes the device of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0069] Figure 3 is a schematic block diagram of an electronic device provided in an embodiment of the present application, and the electronic device may include: The memory 301 and the processor 302, the memory 301 is used to store the computer program and transmit the program code to the processor 302. In other words, the processor 302 can call and run the computer program from the memory 301 to implement the method in the embodiment of the present application.

[0070] For example, the processor 302 may be configured to execute the above method embodiments according to instructions in the computer program.

[0071] In some embodiments of the present application, the processor 302 may include but is not limited to: General-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0072] In some embodiments of the present application, the memory 301 includes but is not limited to: Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct memory bus random access memory (Direct Rambus RAM, DR RAM).

[0073] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 301 and executed by the processor 302 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0074] like Figure 3 As shown, the electronic device may also include: The transceiver 303 may be connected to the processor 302 or the memory 301 .

[0075] The processor 302 may control the transceiver 303 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 303 may include a transmitter and a receiver. The transceiver 303 may further include an antenna, and the number of antennas may be one or more.

[0076] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0077] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0078] When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0079] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0080] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0081] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0082] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for transferring pictures to documents, characterized in that: include: Inputting the first picture into an object detection model to obtain at least one object, wherein the object detection model is used to detect objects of different object types in the picture, and the object types include image types, formula types, and table types; For each object in the at least one object, construct a first correspondence relationship between the object and an object identifier corresponding to the object to obtain at least one first correspondence relationship; Processing the first picture based on at least one object identifier corresponding to the at least one object to obtain a second picture; Recognize the text information in the second picture to obtain a first document; A second document corresponding to the first image is determined based on the first document and the at least one first corresponding relationship.

2. The method according to claim 1, characterized in that The first picture is one of the first pictures obtained by segmenting a complete picture.

3. The method according to claim 2, characterized in that The step of segmenting the complete picture to obtain a plurality of first pictures includes: Taking the first image edge of the complete image as the reference edge; A line segment in the complete image that faces a preset direction, has a preset distance from the reference edge, and has the same length as the reference edge is used as a pending segmentation line; Determine whether the pending segmentation line meets a preset condition, if so, segment the complete image from the pending segmentation line, if not, determine a new pending segmentation line based on a preset moving interval, and segment the complete image from the new pending segmentation line; Determine whether the distance between the second picture edge of the complete picture in the preset direction and the dividing line closest to the second picture edge is less than the preset distance, if so, use the pictures divided at each time as multiple first pictures, if not, use the second picture edge of the complete picture in the preset direction and the dividing line closest to the second picture edge as new reference edges; Return to the step of using the line segment in the complete image that is oriented in the preset direction, has a preset distance from the reference edge, and has the same length as the reference edge as the pending segmentation line, until the distance between the second image edge of the complete image in the preset direction and the segmentation line closest to the second image edge is less than the preset distance.

4. The method according to claim 3, characterized in that The determining of a new pending segmentation line based on a preset moving interval includes: Taking the pending segmentation line as the center, move pixel by pixel to both sides of the pending segmentation line respectively. During the movement, if the number of pixel values ​​of the pending segmentation line at a position that is a first preset value is greater than the first preset number, the pending segmentation line at the position is used as a new pending segmentation line; The first preset value is the pixel value of the background area of ​​the complete image, and when the number of pixel values ​​in the pending segmentation line that are the second preset value is greater than the second preset number, the pending segmentation line is deemed to meet the preset condition.

5. The method according to claim 2, characterized in that: The object type also includes a text type, each first picture corresponds to a second document, and the method further includes: splicing the multiple second documents corresponding to the multiple first images to obtain a first target document corresponding to the complete image; Acquire at least one detection information of at least one object to be detected whose format type is a title format in the first target document, wherein the detection information includes the object type of the object to be detected, the address information of the object to be detected, and the number of text words of the object to be detected whose object type is the text type; Based on the at least one detection information, determining a target object that does not meet a preset title rule from the at least one object to be detected; The title format of the target object in the first target document is cleared to obtain a second target document.

6. The method according to claim 1, characterized in that When there is a formula object of a formula type in the at least one object, before building a first corresponding relationship between the formula object and the formula object, the method further includes: Inputting the formula object into a preset formula recognition model to obtain formula information; Constructing a first correspondence between the formula object and the object identifier corresponding to the formula object, including: A first corresponding relationship between the formula information and the object identifier corresponding to the formula object is constructed.

7. The method according to claim 1, characterized in that The method further comprises: When it is detected that the first number of object identifiers included in the first document is smaller than the second number of object identifiers included in the at least one first corresponding relationship, obtaining the object identifiers to be supplemented that are missing from the first document relative to the at least one first corresponding relationship; The object corresponding to the to-be-supplemented object identifier is supplemented in the second document.

8. A device for transferring pictures to documents, characterized in that: include: An input unit, used to input the first picture into the target detection model to obtain at least one object, wherein the target detection model is used to detect objects of different object types in the picture, and the object types include image types, formula types, and table types; A construction unit, configured to construct, for each object in the at least one object, a first correspondence between the object and an object identifier corresponding to the object, to obtain at least one first correspondence; a processing unit, configured to process the first picture based on at least one object identifier corresponding to the at least one object to obtain a second picture; A recognition unit, used to recognize text information in the second image to obtain a first document; A determination unit is used to determine a second document corresponding to the first image based on the first document and the at least one first corresponding relationship.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images