Electronic device and method for structuring table in electronic document

Abstract

An electronic device according to an embodiment of the present disclosure may: identify, from an electronic document, a plurality of cell boxes constituting a table and at least one text box including text included in the table; allocate the at least one text box to at least one cell box among the plurality of cell boxes, on the basis of coordinate information about the plurality of cell boxes and coordinate information about the at least one text box; identify at least one boundary line for distinguishing rows and / or columns of the table, on the basis of a plurality of X coordinates and a plurality of Y coordinates included in the coordinate information about the plurality of cell boxes; correct the coordinate information about the plurality of cell boxes on the basis of the at least one boundary line; and reallocate the at least one text box to the plurality of cell boxes on the basis of the corrected coordinate information about the plurality of cell boxes.

Description

Electronic device and method for structuring a table in an electronic document

[0001] The present disclosure relates to an electronic device and method for structuring a table in an electronic document.

[0002] Electronic documents can take a digital format on electronic devices such as computers or mobile devices. Electronic documents can be edited or read on electronic devices and can contain various content, such as text, images, or tables. In electronic documents, tables can be used to structurally display a summary of information or data. Generally, tables are composed of rows and columns, which can enable the systematic organization of information or data and facilitate easy visual analysis.

[0003] Recently, methods for recognizing tables in electronic documents have been actively researched. While table recognition methods are continuously evolving through various technologies, no methods have been proposed to measure or provide the reliability of the recognition results. For recognized tables to be used for important decision-making or widely utilized in various fields, reliability—indicating how accurate and trustworthy the recognized table is—is required as an important factor.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of this document. None of the foregoing is to be claimed as prior art related to this document, nor is it to be used to determine prior art.

[0005] One embodiment of the present disclosure may provide an electronic device and method for structuring a table in an electronic document.

[0006] One embodiment of the present disclosure may provide an electronic device and method for measuring the reliability of a table recognized (identified or determined) in an electronic document.

[0007] One embodiment of the present disclosure may provide an electronic device and method for providing structural reliability based on the structuring of a table in an electronic document.

[0008] One embodiment of the present disclosure may provide an electronic device and method for providing content reliability based on the result of table recognition (identification or determination) in an electronic document.

[0009] An electronic device according to one embodiment of the present disclosure may include a memory and at least one processor that executes instructions of a program stored in the memory. The at least one processor may identify a plurality of cell boxes constituting a table and at least one text box containing text included in the table from an electronic document, assign the at least one text box to at least one of the plurality of cell boxes based on the coordinate information of the plurality of cell boxes and the coordinate information of the at least one text box, identify at least one boundary line for distinguishing at least one row or column of the table based on a plurality of X coordinates and a plurality of Y coordinates included in the coordinate information of the plurality of cell boxes, correct the coordinate information of the plurality of cell boxes based on the at least one boundary line, and reassign the at least one text box to the plurality of cell boxes based on the corrected coordinate information of the plurality of cell boxes.

[0010] According to one embodiment, the at least one text box represents at least one box area for identifying the boundaries of text included in the table, and the plurality of cell boxes may represent box areas for identifying the boundaries of a plurality of cells constituting the table.

[0011] According to one embodiment, the text included in the at least one text box may include at least one of a word, a sentence including spaces, or a paragraph including a line break.

[0012] According to one embodiment, the at least one processor can assign at least one text box closest to the at least one cell box to the at least one cell box.

[0013] According to one embodiment, the at least one processor can obtain information about the at least one boundary line using a pre-trained classification model based on the plurality of X coordinates and the plurality of Y coordinates.

[0014] According to one embodiment, the at least one processor can correct the plurality of X coordinates so that the plurality of X coordinates are aligned based on the first boundary line, based on the fact that the at least one boundary line includes a first boundary line for separating the columns of the table, and update the coordinate information of the plurality of cell boxes so that information about the plurality of corrected X coordinates is included.

[0015] According to one embodiment, the at least one processor can correct the plurality of Y coordinates so that the plurality of Y coordinates are aligned based on the second boundary line, based on the fact that the at least one boundary line includes a second boundary line for separating the rows of the table, and update the coordinate information of the plurality of cell boxes so that information about the plurality of corrected Y coordinates is included.

[0016] According to one embodiment, the at least one processor can correct the plurality of X coordinates so that the plurality of X coordinates are aligned based on the first boundary line, based on the at least one boundary line including a first boundary line for separating the columns of the table, correct the plurality of Y coordinates so that the plurality of Y coordinates are aligned based on the second boundary line, based on the at least one boundary line including a second boundary line for separating the rows of the table, and update the coordinate information of the plurality of cell boxes so that the information regarding the plurality of corrected X coordinates and the information regarding the plurality of corrected Y coordinates are included.

[0017] According to one embodiment, the electronic device may further include a display. The at least one processor may update the table based on reassigning the at least one text box to the plurality of cell boxes, obtain a first result that recognizes the whole of the updated table and a second result that recognizes a part of the updated table, determine the reliability of the updated table based on the degree of similarity between the part corresponding to the second result within the first result and the second result, and display information about the determined reliability on the display.

[0018] According to one embodiment, the at least one processor may display information on the display indicating that a part of the updated table has a reliability level below the threshold value, based on the determined reliability level being below the threshold value.

[0019] A method of operation of an electronic device according to one embodiment of the present disclosure may include: identifying at least one text box containing text included in a plurality of cell boxes constituting a table from an electronic document; assigning the at least one text box to at least one of the plurality of cell boxes based on the coordinate information of the plurality of cell boxes and the coordinate information of the at least one text box; identifying at least one boundary line for distinguishing at least one row or column of the table based on a plurality of X coordinates and a plurality of Y coordinates included in the coordinate information of the plurality of cell boxes; correcting the coordinate information of the plurality of cell boxes based on the at least one boundary line; and reassigning the at least one text box to the plurality of cell boxes based on the corrected coordinate information of the plurality of cell boxes.

[0020] According to one embodiment, the at least one text box represents at least one box area for identifying the boundaries of text included in the table, and the plurality of cell boxes may represent box areas for identifying the boundaries of a plurality of cells constituting the table.

[0021] According to one embodiment, the text included in the at least one text box may include at least one of a word, a sentence including spaces, or a paragraph including a line break.

[0022] According to one embodiment, the operation of assigning the at least one text box may include the operation of assigning at least one text box closest to the at least one cell box among the at least one text boxes to the at least one cell box.

[0023] According to one embodiment, the operation of identifying the at least one boundary line may include: the operation of obtaining information about the at least one boundary line using a pre-trained classification model based on the plurality of X coordinates and the plurality of Y coordinates; and the operation of identifying the at least one boundary line based on the obtained information.

[0024] According to one embodiment, the operation of correcting the coordinate information of the plurality of cell boxes may include: an operation of correcting the plurality of X coordinates so that the plurality of X coordinates are aligned based on the first boundary line, based on the fact that the at least one boundary line includes a first boundary line for distinguishing the columns of the table; and an operation of updating the coordinate information of the plurality of cell boxes so that information regarding the plurality of corrected X coordinates is included.

[0025] According to one embodiment, the operation of correcting the coordinate information of the plurality of cell boxes may include: an operation of correcting the plurality of Y coordinates so that the plurality of Y coordinates are aligned based on the second boundary line, based on the fact that the at least one boundary line includes a second boundary line for distinguishing the rows of the table; and an operation of updating the coordinate information of the plurality of cell boxes so that information regarding the plurality of corrected Y coordinates is included.

[0026] According to one embodiment, the operation of correcting the coordinate information of the plurality of cell boxes may include: an operation of correcting the plurality of X coordinates so that the plurality of X coordinates are aligned based on the first boundary line, based on the fact that the at least one boundary line includes a first boundary line for distinguishing the columns of the table; an operation of correcting the plurality of Y coordinates so that the plurality of Y coordinates are aligned based on the second boundary line, based on the fact that the at least one boundary line includes a second boundary line for distinguishing the rows of the table; and an operation of updating the coordinate information of the plurality of cell boxes so that information regarding the plurality of corrected X coordinates and information regarding the plurality of corrected Y coordinates are included.

[0027] According to one embodiment, the operation method may further include: an operation of updating the table based on reassigning the at least one text box to the plurality of cell boxes; an operation of obtaining a first result of recognizing the whole of the updated table and a second result of recognizing a part of the updated table; an operation of determining the reliability of the updated table based on the degree of similarity between the part corresponding to the second result within the first result and the second result; and an operation of displaying information about the determined reliability on the display.

[0028] According to one embodiment, the method may further include an operation of displaying information on the display indicating that a part of the updated table has a reliability level below the threshold value, based on the determined reliability level being below the threshold value.

[0029] FIG. 1 is a block diagram of an electronic device according to one embodiment.

[0030] FIG. 2a is a drawing illustrating a table structuring device according to one embodiment.

[0031] FIG. 2b is a diagram illustrating coordinate information of each cell box and text box according to one embodiment.

[0032] FIG. 3 is a diagram illustrating the operation of a table structuring module according to one embodiment.

[0033] FIG. 4a is a drawing illustrating a table included in an electronic document according to one embodiment.

[0034] FIG. 4b is a drawing illustrating text boxes and cell boxes according to one embodiment.

[0035] FIG. 4c is a diagram illustrating a table structuring operation according to one embodiment.

[0036] FIG. 4d is a drawing illustrating a structured table according to one embodiment.

[0037] FIG. 5 is a diagram illustrating a table structured by a table structuring operation according to one embodiment.

[0038] FIG. 6 is a diagram illustrating the operation of a sub-table comparison module according to one embodiment.

[0039] FIG. 7 is a diagram illustrating the operation of recognizing a table and generating a tree structure according to one embodiment.

[0040] FIG. 8 is a diagram illustrating the operation of measuring the similarity of two tables according to one embodiment.

[0041] FIG. 9 is a flowchart illustrating the table structuring operation of an electronic device according to one embodiment.

[0042] FIG. 10 is a flowchart illustrating the operation of an electronic device according to one embodiment updating coordinate information of a plurality of cell boxes.

[0043] FIG. 11 is a flowchart illustrating the reliability provision operation of an electronic device according to one embodiment.

[0044] In the following description, the attached drawings are referenced, and specific examples of implementation are illustrated within the drawings. Additionally, other examples may be used and structural modifications may be made without departing from the scope of the various examples.

[0045] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0046] FIG. 1 is a block diagram of an electronic device according to one embodiment.

[0047] Referring to FIG. 1, the electronic device (100) may include a display (110), memory (120), and a processor (130). According to one embodiment, the electronic device (100) may include additional components (e.g., a user interface or a transceiver) in addition to the illustrated components, or at least one of the illustrated components may be omitted. According to one embodiment, the electronic device (100) may be any one of a mobile device (e.g., a smartphone or tablet), a computing device (e.g., a PC (personal computer) or a laptop), a wearable device (e.g., a smart watch or HMD (head mounted display)), or a home appliance (e.g., a TV (television)), but is not limited thereto and may be various types of electronic devices.

[0048] According to one embodiment, the display (110) can perform various display operations according to the function of the electronic device (100). For example, the display (110) can display various forms of information such as numbers, characters, images, graphics, or tables. The display (110) may be configured as a touchscreen by forming a layer structure with a touch pad. In this case, the display (110) may also be used as an input interface in addition to an output interface. The display (110) may be a single independent display or may include multiple displays. Multiple displays may be placed at different locations.

[0049] According to one embodiment, the memory (120) may store various data used by at least one component of the electronic device (100) (e.g., a display (110) or a processor (130)). For example, the memory (120) may store at least one program for processing and controlling the processor (130) and may store input and / or output data. According to one embodiment, the memory (120) may store an artificial intelligence (AI) model (or a machine learning model or a deep learning model) and may store data or information learned through the AI ​​model. According to one embodiment, the memory (120) may include volatile memory or non-volatile memory. According to one embodiment, a web storage or cloud server that performs storage functions over the Internet may be operated by the electronic device (100).

[0050] According to one embodiment, the processor (130) can control the overall operation of the electronic device (100). The processor (130) can perform operations or data processing regarding the control and / or communication of at least one other component of the electronic device (100). For example, the processor (130) can be electrically connected to the display (110) and the memory (120) and can execute instructions of a program stored in the memory (120).

[0051] The processor (130) may include a processing circuit that executes instructions of a program stored in memory (120). The processor (130) may include at least one of a CPU (central processing unit), NPU (neural processing unit), GPU (graphics processing unit), MPU (micro processing unit), MCU (micro controller unit), AP (application processor), CP (communication processor), SoC (system on chip), IC (integrated circuit) sensor hub, supplementary processor, ASIC (application specific integrated circuit), or FPGA (field programmable gate arrays), and may have multiple cores.

[0052] The processor (130) can control the operations of the electronic device (100) by executing instructions stored in the memory (120). For example, the processor (130) may correspond to a plurality of processors that collectively perform a plurality of operations by dividing (or allocating) them among the processors. According to one embodiment, the processor (130) can perform the operations of the electronic device (100) described below.

[0053] FIG. 2a is a drawing illustrating a table structuring device according to one embodiment.

[0054] Referring to FIG. 2a, the table structuring device (200) may be included in the processor (130) of the electronic device (100), be a component corresponding to the processor (130), or be an independent component electrically connected to the processor (130) and operating based on the control of the processor (130). According to one example, the table structuring device (200) may be computer code, computer codes, computer programs, or computer programs stored or included in memory (120). In the present disclosure, the table structuring device (200) may be a table structure code, table structure codes, table structure programs, or table structure programs.

[0055] According to one embodiment, the table structuring device (200) may include a table recognition module (204), a table structuring module (230), and a sub-table comparison module (240).

[0056] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0057] In the present disclosure, the table recognition module (204) may be a table recognition code, table recognition codes, a table recognition program (software), or a table recognition processor (hardware). The table structuring module (230) may be a table structuring code, table structuring codes, a table structuring program (software), or a table structuring processor (hardware). The sub-table comparison module (240) may be a sub-table comparison code, sub-table comparison codes, a sub-table comparison program (software), or a sub-table comparison processor (hardware).

[0058] According to one embodiment, the table recognition module (204) may receive or input an electronic document (202). For example, the electronic document (202) may be an electronic document in image format and may include at least one table. The electronic document (202) may be selected by a user from among the electronic documents stored in the electronic device (100). According to one aspect of the present disclosure, the electronic document (202) may be received from an external electronic device.

[0059] According to one embodiment, the table recognition module (204) may use an AI model that has been learned or trained for table recognition (identification or determination) based on (using) an electronic document (202) input. The AI ​​model may be included in an electronic device (100) or in at least one server on a network. For example, the AI ​​model may be a neural network model and may include a text recognition model (210) that recognizes (identifies or determines) text contained in the table and a table structure recognition model (220) that recognizes (identifies or determines) the structure of the table. The text recognition model (210) and the table structure recognition model (220) may be composed of separate models as shown in FIG. 2, but may also be composed of a single integrated model.

[0060] According to one embodiment, the text recognition model (210) may represent a model trained to recognize (identify or determine) text contained in an image. For example, the text recognition model (210) may include an optical character recognition (OCR) model used to extract or recognize visual forms of text from an image.

[0061] According to one embodiment, the table recognition module (204) can identify text content (212) and a text box (214) included in an electronic document (202) based on a text recognition model (210). The text content (212) may represent text included in a table of the electronic document (202). The text box (214) may represent a box area for identifying the boundaries of the text included in the table.

[0062] According to one embodiment, the table structure recognition model (220) may represent a model trained to recognize (identify or determine) structural features of a table included in an image. The table structure recognition model (220) may be used to identify at least one cell constituting the table, or rows and columns of the table.

[0063] According to one embodiment, the table recognition module (204) can obtain cell boxes (222) and table structuring information (e.g., a Hypertext Markup Language (HTML) document, hereinafter referred to as 'HTML document') (224) included in an electronic document (202) based on a table structure recognition model (220).

[0064] According to one embodiment, the cell box (222) may represent a box area for determining the boundaries of the cells constituting the table. The cell box (222) may include one or more text boxes, or may not include any text boxes.

[0065] According to one embodiment, an HTML document (224) may represent a document that defines each cell of a table, or each row and column of a table, and structurally represents the text contained in each cell of the table, or each row and column of the table, using HTML tags. For example, HTML tags , , or It may include. It can define the start and end of the table. silver It is located inside and can represent the rows of the table. Is It is located inside and can represent a table cell containing text. , , or At least one of them may include various attribute values ​​that indicate the structure or layout of the table. For example,

[0126]

[0127]

[0128]

[0129]

[0130]

[0131]

[0132]

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140]

[0141]

[0142]

[0143]

[0144]

[0145] It may include at least one of the following as an attribute value: border, which specifies the thickness of the table border; cellpadding, which specifies the margin between the boundary and content of each cell; cellspacing, which specifies the spacing between each cell; width, which specifies the width of the table; or height, which specifies the height of the table. For example, It can include bgcolor as an attribute value that specifies the background color of the row. For example, It may include a colspan that indicates how many columns the cell extends to, or a rowspan that indicates how many rows the cell extends to.

[0066] According to one embodiment, a table structuring module (230) may be used to obtain structure confidence (or structural confidence) (234) for a recognized table, and a sub-table comparison module (240) may be used to obtain contents confidence (242). According to one embodiment, structure confidence (234) may indicate the degree or similarity with which the structural features of the recognized table match the structural features of the table within the electronic document (202). According to one embodiment, contents confidence (242) may indicate the degree or similarity with which the text content included in the recognized table matches the text content included in the table within the electronic document (202). Higher structure confidence (234) and contents confidence (242) may indicate that the table recognition is performed accurately.

[0067] According to one embodiment, information regarding a text box (214) obtained based on a text recognition model (210) and a cell box (222) obtained based on a table structure recognition model (220) may be input to the table structuring module (230). The table structuring module (230) may perform alignment on the text box (214) and the cell box (222) and generate a grid box (232) of a grid structure that distinguishes cells, rows, or columns within the table based on the alignment result. The table structuring module (230) may use the alignment result as an indicator of the structural reliability (234) of the table.

[0068] According to one embodiment, the sub-table comparison module (240) may receive text content (212) obtained based on a text recognition model (210) and an HTML document (224) obtained based on a table structure recognition model (220). The sub-table comparison module (240) may be a module that can be used optionally, and the consistency between the result of recognizing the entire table within the electronic document (202) and the result of recognizing a portion of the table cut out may be used as an indicator of content reliability (242).

[0069] According to one embodiment, the table structuring module (230) and the sub-table comparison module (240) may each be configured independently or configured as a single integrated module to be used together with the table recognition module (204) (or the text recognition model (210) and the table structure recognition model (220)).

[0070] According to one embodiment, the table structuring module (230) and / or the sub-table comparison module (240) may be used optionally. For example, if the structural reliability (234) of the table is not required, the table structuring module (230) may not be used, and if the content reliability (242) is not required, the sub-table comparison module (240) may not be used. To increase the table recognition speed, the table structuring module (230) and / or the sub-table comparison module (240) may not be used.

[0071] FIG. 2b is a diagram illustrating coordinate information of each cell box and text box according to one embodiment.

[0072] Referring to FIG. 2b, the cell box (222) may represent a box area for determining the boundaries of cells constituting a table. The cell box (222) may include one or more text boxes (e.g., text box (214)) or may not include any text boxes. According to one embodiment, the recognized location of the cell box (222) within the electronic document (202) may be indicated by coordinate information. The coordinate information of the cell box (222) may include coordinate information associated with a two-dimensional box area and may have various forms.

[0073] For example, the coordinate information of the cell box (222) may include (x1, y1), (x1, y2), (x2, y1), and (x2, y2) as coordinate information corresponding to each vertex of the cell box (222).

[0074] For example, the coordinate information of the cell box (222) may include coordinate information in the form of (x1, y1, w, h) based on a reference point (or starting point) (x1, y1) within the cell box (222). W represents a distance from x1 on the X-axis and can indicate the width of the cell box (222) and can be used to infer x2. H represents a distance from y1 on the Y-axis and can indicate the height of the cell box (222) and can be used to infer y2.

[0075] For example, the coordinate information of the cell box (222) may include coordinate information in the form of (x1, y1, x2, y2). (x1, y1, x2, y2) may indicate two coordinates (x1, x2) on the X-axis and two coordinates (y1, y2) on the Y-axis that constitute the cell box (222).

[0076] According to one embodiment, the coordinate information of the cell box (222) may further include information about rows and columns. For example, when information about rows and / or columns corresponding to each of the cell boxes (222) is output through the table structure recognition model (220) of the neural network, information about rows and columns may be added to the coordinate information, such as ([x1, y1, x2, y2], row, column).

[0077] According to one embodiment, the text box (214) may represent a box area for identifying the boundaries (e.g., top, bottom, left, and right boundaries) of text (e.g., OCTOBER). The text included in the text box (214) or the text included in the cell constituting the table may include at least one of a word, a sentence including spaces, or a paragraph including a line break.

[0078] According to one embodiment, the recognized location of a text box (214) within an electronic document (202) may be indicated by coordinate information. The coordinate information of the text box (214) may include coordinate information associated with a two-dimensional box area and may have various forms.

[0079] For example, the coordinate information of the text box (214) may include (x'1, y'1), (x'1, y'2), (x'2, y'1), and (x'2, y'2) as coordinate information corresponding to each vertex of the text box (214).

[0080] For example, the coordinate information of the text box (214) may include coordinate information in the form of (x'1, y'1, w', h') based on a reference point (or starting point) (x'1, y'1) within the text box (214). w' represents a distance from x'1 on the X-axis and can indicate the width of the cell box (222) and can be used to infer x'2. h' represents a distance from y'1 on the Y-axis and can indicate the height of the text box (214) and can be used to infer y'2.

[0081] For example, the coordinate information of the text box (214) may include coordinate information in the form of (x'1, y'1, x'2, y'2). (x'1, y'1, x'2, y'2) may indicate two coordinates (x'1, x'2) on the X-axis and two coordinates (y'1, y'2) on the Y-axis that constitute the text box (214).

[0082] According to one embodiment, the coordinate information of the cell box (222) may include (x0, y0), which is the center coordinate information or the coordinate information corresponding to the zero point, and the coordinate information of the text box (214) may include (x'0, y'0), which is the center coordinate information or the coordinate information corresponding to the zero point. (x0, y0) and (x'0, y'0) can be used to measure the distance between the cell box (222) and the text box (214) and to assign the text box (214) to the cell box (222). For example, if a plurality of text boxes adjacent to the cell box (222) are recognized, the text box (e.g., text box (214)) that has the shortest distance from the cell box (222) among the plurality of text boxes may be assigned to the cell box (222). According to one embodiment, the distance between the cell box (222) and the text box (214) may be measured based on other coordinate information (e.g., (x1, y1) and (x'1, y'1)) in addition to (x0, y0) and (x'0, y'0).

[0083] The operation of the table structuring module (230) of FIG. 2a will be explained in detail below with reference to FIG. 3.

[0084] FIG. 3 is a diagram illustrating the operation of a table structuring module according to one embodiment.

[0085] Referring to FIG. 3, the table structuring module (230) can identify text boxes (302) and cell boxes (304) in an electronic document (202) based on information output from the table recognition module (204). According to one embodiment, there may be one or more text boxes (302) and multiple cell boxes. Hereinafter, the text boxes (302) will be referred to as one or more text boxes (302), and the cell boxes (304) will be referred to as cell boxes (304).

[0086] In operation 310, the table structuring module (230) may assign a text box to each cell box based on the identification of one or more text boxes (302) and cell boxes (304). According to one embodiment, one or more text boxes (302) may include the text boxes (214) of FIGS. 2a and FIGS. 2b, and cell boxes (304) may include the cell boxes (222) of FIGS. 2a and FIGS. 2b.

[0087] According to one embodiment, the table structuring module (230) can calculate the distance between each text box and each cell box based on the coordinate information of each of one or more text boxes (302) and the coordinate information of each of the cell boxes (304). For example, the coordinate information of each of one or more text boxes (302) may be the coordinate information constituting each text box (e.g., the coordinate information corresponding to the zero point of each text box), and the coordinate information of each of the cell boxes (304) may be the coordinate information constituting each cell box (e.g., the coordinate information corresponding to the zero point of each cell box). Based on the calculated distance, the table structuring module (230) can identify at least one text box closest to at least one cell box among one or more text boxes (302) and assign the identified at least one text box to at least one cell box. Based on the assignment of at least one text box to at least one cell box, a cell-specific text box group (312) may be formed.

[0088] According to one embodiment, the table structuring module (230) can collect coordinates for cell boxes (304) based on the formation of cell-specific text box groups (312). For example, the table structuring module (230) can perform operations 320 and 330.

[0089] In operation 320, the table structuring module (230) can collect the x-coordinates of the cell boxes (304) in column units based on the coordinate information and row / column information (e.g., ([x1, y1, x2, y2], row, column)) of each cell box (304) output through the table structure recognition model (220). In operation 322, the table structuring module (230) can obtain x-coordinate groups classified in column units.

[0090] In operation 330, the table structuring module (230) can collect the y-coordinates of the cell boxes (304) on a row-by-row basis based on the coordinate information and row / column information (e.g., ([x1, y1, x2, y2], row, column)) of each cell box (304) output through the table structure recognition model (220). In operation 332, the table structuring module (230) can obtain groups of y-coordinates classified on a row-by-row basis. Operations 320 and 330 may be performed simultaneously, or operation 330 may be performed prior to operation 320.

[0091] In operation 340, the table structuring module (230) can generate a decision boundary (hereinafter referred to as a “column boundary”) (342) for distinguishing neighboring columns in a group of x-coordinates (322) classified by column, based on a classification algorithm or a classification model (e.g., a support vector machine (SVM)). For example, the table structuring module (230) can generate a column boundary (342) for distinguishing neighboring columns based on the x-coordinates of each neighboring column. The table structuring module (230) can correct the x-coordinates of each neighboring column so that the neighboring columns are aligned based on the column boundary (342). For example, the table structuring module (230) can correct the x-coordinates of each neighboring column so that the x-coordinates of each neighboring column correspond to the x-coordinates of the column boundary (342). In this case, each side of the adjacent cell boxes may have an x-coordinate corresponding to the column boundary line (342), and an improved alignment accuracy (344) in the column direction may be obtained.

[0092] In operation 350, the table structuring module (230) may generate a y-coordinate group (322) classified by row based on a classification algorithm or classification model (e.g., SVM) to distinguish adjacent rows. For example, the table structuring module (230) may generate a boundary line (hereinafter referred to as a ‘row boundary line’) (354) to distinguish adjacent rows based on the y-coordinate of each adjacent row. The table structuring module (230) may correct the y-coordinate of each adjacent row so that the adjacent rows are aligned based on the row boundary line (354). For example, the table structuring module (230) may correct the y-coordinate of each adjacent column so that the y-coordinate of each adjacent row corresponds to the y-coordinate of the row boundary line (354). In this case, each side of the adjacent cell boxes may have a y-coordinate corresponding to the row boundary line (354), and improved alignment accuracy (352) in the row direction may be obtained.

[0093] According to one embodiment, the table structuring module (230) can calculate structural reliability (356) based on column alignment accuracy (344) and row alignment accuracy (352). Structural reliability (356) may indicate the probability that the structural features of the recognized table match the structural features of the table within the electronic document (202), and may indicate that the higher the structural reliability (356), the more accurately the table recognition is performed. According to one embodiment, structural reliability (356) may be calculated as shown in the following [Equation 1].

[0094]

[0095] Referring to [Equation 1], i can represent a column index and j can represent a row index. can represent the sorting accuracy of the i-th column, and can represent the alignment accuracy of the j-th row. Is and Based on (or and It can indicate the structural reliability of the table (determined by the product of ).

[0096] In operation 360, the table structuring module (230) may reassign text boxes to each cell box based on boundaries (e.g., column boundaries (342) and / or row boundaries (354)). For example, the table structuring module (230) may identify at least one text box closest to each of the positionally corrected or aligned cell boxes (304) based on the correction of the x and y coordinates of each of the cell boxes (304), and assign the identified at least one text box to the corresponding cell box. The table structuring module (230) may generate a table containing the structured cell boxes (304) by assigning at least one text box to each of the aligned cell boxes (304). The table structuring module (230) may display the generated table on a display or repeat the operations of operation 302 and below.

[0097] According to one embodiment, the table structuring module (230) may repeat operations 302 or lower until the structural reliability (356) converges to a set reliability (e.g., until it exceeds a threshold), or repeat operations 302 or lower a preset number of times (e.g., N times).

[0098] Referring to FIGS. 4a to 4d below, the operation of obtaining a structured table will be explained.

[0099] FIG. 4a is a drawing illustrating a table included in an electronic document according to one embodiment.

[0100] Referring to FIG. 4a, an electronic document (202) may be input into a table structuring device (200), and the input electronic document (202) may include a table (400). According to one embodiment, the table included in the electronic document (202) may be one or more, and may include a table having a different shape, different design, different size, or different number of rows / columns from the table (400) shown in FIG. 4a.

[0101] According to one embodiment, the table structuring device (200) can identify text boxes and cell boxes included in the table (400) based on the input of an electronic document (202), based on a text recognition model (e.g., an OCR model) (210) and a table structure recognition model (220), respectively. For example, the table structuring device (200) can identify text boxes and cell boxes as shown in FIG. 4b.

[0102] FIG. 4b is a drawing illustrating text boxes and cell boxes according to one embodiment.

[0103] Referring to FIG. 4b, the table structuring device (200) can identify text boxes included in the table (400) based on a text recognition model (210). Each of the text boxes may be a box area for identifying the boundaries of text, and may contain text in units of words, sentences, or paragraphs, or may contain spaces or whitespace (e.g., whitespace for line breaks).

[0104] The table structuring device (200) can identify cell boxes included in the table (400) based on the table structure recognition model (220). Each cell box is a box area for identifying the boundary of a cell constituting the table, and may include one or more text boxes or not include any text boxes.

[0105] The text boxes and cell boxes shown in FIG. 4b may have an unaligned form. The table structuring device (200) may perform a table structuring operation as shown in FIG. 4c so that the text boxes and cell boxes may have an aligned and structured form.

[0106] FIG. 4c is a diagram illustrating a table structuring operation according to one embodiment.

[0107] Referring to FIG. 4c, the table structuring operation can be performed by the table structuring module (230) of the table structuring device (200). The table structuring module (230) obtains coordinate information and row / column information (e.g., ([x1, y1, x2, y2], row, column)) of each cell box through the table structure recognition model (220), and can group the x coordinates and y coordinates of the cell boxes into column units and row units, respectively, based on the obtained information.

[0108] The table structuring module (230) can generate column boundaries to distinguish neighboring columns in groups of x-coordinates classified by column, based on a classification algorithm or classification model (e.g., SVM). For example, the table structuring module (230) can generate column boundaries (412) to distinguish the x-coordinates of neighboring columns in a first part (410) of the table (400).

[0109] The table structuring module (230) can correct the x-coordinates to the left and to the right of the column boundary (412). For example, the table structuring module (230) can align cell boxes to positions based on the column boundary (412) by correcting the x-coordinates to the left and to the right of the column boundary (412) to correspond to the x-coordinates of the column boundary (412). The table structuring module (230) can create additional column boundaries for parts other than the first part (410) in a manner similar to the above, and can perform alignment operations on the additionally created column boundaries as well. Accordingly, cell boxes can be aligned in the column direction.

[0110] The table structuring module (230) can generate row boundaries to distinguish neighboring rows in groups of y coordinates classified by row, based on a classification algorithm or classification model (e.g., SVM). For example, the table structuring module (230) can generate row boundaries (422) to distinguish between upper y coordinates and lower y coordinates in the second part (420) of the table (400).

[0111] The table structuring module (230) can correct the y-coordinates above and below the row boundary line (422). For example, the table structuring module (230) can align cell boxes to a position based on the row boundary line (422) by correcting the y-coordinates above and below the row boundary line (422) to correspond to the y-coordinates of the row boundary line (422). The table structuring module (230) can create additional row boundary lines for the second part (420) and other parts in a manner similar to the above, and can perform alignment operations on the additionally created row boundary lines as well. Accordingly, the cell boxes can be aligned in the row direction.

[0112] The table structuring module (230) can perform reassignment of text boxes based on the cell boxes being aligned in the column direction and / or row direction. For example, the table structuring module (230) can identify at least one text box closest to each of the aligned cell boxes and reassign the identified at least one text box to the corresponding cell box. According to one embodiment, the reassignment result may be as shown in FIG. 4d.

[0113] FIG. 4d is a drawing illustrating a structured table according to one embodiment.

[0114] Referring to FIG. 4d, the table structuring module (230) can generate a structured table (410) by displaying a grid box of a grid structure that separates cells, rows, or columns based on the completion of alignment operations for cell boxes and reassignment operations for text boxes.

[0115] The structured table (410) illustrated in FIG. 4d has an aligned form compared to the table (400) in FIG. 4b, so the structural reliability can be high and it can be made possible to analyze data more clearly visually.

[0116] FIG. 5 is a diagram illustrating a table structured by a table structuring operation according to one embodiment.

[0117] Referring to FIG. 5(a), the table (500) recognized from an electronic document by the table structuring device (200) may have some cell boxes (510) inaccurately recognized, and text boxes may be incorrectly assigned. For example, no text boxes may be assigned to a single cell box, or multiple text boxes may be assigned to a single cell box. In this case, the accuracy or reliability of the table (500) may be reduced because data matching the items is not mapped. To prevent this, the table structuring device (200) may use a table structuring module (230). According to one embodiment, the table structuring module (230) may perform a table structuring operation on the table (500) to generate a structured table (520) as shown in FIG. 5(b).

[0118] Referring to FIG. 5(b), the table structuring module (230) can generate a structured table (520) by aligning cell boxes (e.g., column and / or row alignment) and reassigning text boxes to the aligned cell boxes through a table structuring operation. When the table structuring operation is performed, some cell boxes (510) in the table (500) may be corrected and text boxes reassigned as some cell boxes (530) of the structured table (520).

[0119] Compared to the table (500) illustrated in FIG. 5 (a), text boxes can be accurately assigned to some cell boxes (530) of the structured table (520) for each cell box. Accordingly, data matching the items is accurately mapped, and the accuracy or reliability of the structured table (520) can be improved.

[0120] After the table structuring operation described above is performed, a table recognition result comparison operation to measure content reliability may be performed. The table recognition result comparison operation may be performed by the sub-table comparison module (240) described in FIG. 2a. The operation of the sub-table comparison module will be explained below with reference to FIG. 6.

[0121] FIG. 6 is a diagram illustrating the operation of a sub-table comparison module according to one embodiment.

[0122] Referring to FIG. 6, the sub-table comparison module (240) described in FIG. 2a may be optionally used and may perform an operation to compare the results of recognizing a table (602) within an electronic document (202) input to the table recognition module (204). According to one embodiment, the sub-table comparison module (240) may compare the result of the table recognition module (204) recognizing the entire table (602) with the result of the table recognition module (204) recognizing a part of the table (602).

[0123] The sub-table comparison module (240) can obtain cell box information (622) and the first HTML document (624) as a result of recognizing the entire table (602) through the table recognition module (204).

[0124] Cell box information (622) may include coordinate information of each cell included in the table (602). For example, if there are N cell boxes included in the table (602) (N is a natural number greater than or equal to 2), the N cell boxes may be indicated as bbox 1 to bbox N, and the N cell boxes may correspond to coordinate information of {x1, y1, x2, y2}. Here, x1, y1, x2, and y2 indicate two coordinates on the X-axis and two coordinates on the Y-axis constituting each cell box, and may be set to different values ​​for each cell box.

[0125] The first HTML document (624) contains tags that indicate the overall structure of the table (602) (e.g., , , , , or It may include ) and the entire text (e.g., A, B, C, 쪋, N). The sub-table comparison module (240) may obtain at least one HTML document for comparison with the first HTML document (624) based on cell box information (622) output from the table recognition module (204). For example, the sub-table comparison module (240) may obtain the first part (602-1) of the table (602) by performing a first crop operation (610) to cut out a part of the table (602) based on the cell box information (622). The sub-table comparison module (240) can obtain a second HTML document (611) which is the result of recognizing a first part (602-1) through the table recognition module (204). For example, the sub-table comparison module (240) can obtain a second part (602-2) of the table (602) by performing a second crop operation (620) that cuts out a part of the table (602) based on cell box information (622). The sub-table comparison module (240) can obtain a third HTML document (612) which is the result of recognizing the second part (602-2) through the table recognition module (204). For example, the sub-table comparison module (240) can obtain a third part (602-3) of the table (602) by performing a third crop operation (630) that cuts out a part of the table (602) based on cell box information (622). The sub-table comparison module (240) can obtain a fourth HTML document (613) which is the result of recognizing a third part (602-3) through the table recognition module (204). According to one embodiment, the sub-table comparison module (240) can compare each of the second HTML document (611), the third HTML document (612), or the fourth HTML document (613) corresponding to the result of recognizing a part of the table (602) with the first HTML document (624) corresponding to the result of recognizing the whole of the table (602).For example, the sub-table comparison module (240) can detect a first similarity (s1) by comparing the second HTML document (611) with the part (624-1) corresponding to the second HTML document (611) in the first HTML document (624). For example, the sub-table comparison module (240) can detect a second similarity (s2) by comparing the third HTML document (612) with the part (624-2) corresponding to the third HTML document (612) in the first HTML document (624). For example, the sub-table comparison module (240) can detect a third similarity (s3) by comparing the fourth HTML document (613) with the part (624-3) corresponding to the fourth HTML document (613) in the first HTML document (624). According to one embodiment, the sub-table comparison module (240) can, based on the detected similarity, Content reliability can be measured as shown in [Equation 2]. In [Equation 2], represents the number of comparisons for obtaining similarity (e.g., the number of comparisons between the entire table and a part of the table), represents a cell in the i-th column and j-th row, represents a part of the table, represents similarity associated with the part of the table, and represents content reliability. According to one embodiment, the reliability of the table may also be measured as in [Equation 3] based on the structural reliability obtained based on [Equation 1] and the content reliability obtained based on [Equation 2]. In [Equation 3], represents the reliability of the table determined based on , which represents structural reliability, and , which represents content reliability. According to one embodiment, similarity (e.g., first to third similarity) indicates how similar the structure and content (e.g., text or text content) of two tables are, and can be obtained based on various measurement methods.For example, similarity can be obtained based on a similarity measurement method such as TEDS (table edit distance based similarity). A similarity measurement method using TEDS will be described below with reference to FIGS. 7 and 8. FIG. 7 is a diagram illustrating the operation of recognizing a table and generating a tree structure according to one embodiment. According to one embodiment, a table recognition module (204) can recognize a table (710) shown in FIG. 7 (a) from an electronic document and output an HTML document (or HTML code) (720) as shown in FIG. 7 (b). For example, the HTML document (720) may include tags indicating the cell structure of each row included in the table (710) and text included in the cells of each row. The table recognition module (204) can input the HTML document (720) into a sub-table comparison module (240). According to one embodiment, the sub-table comparison module (240) can generate a hierarchical tree structure (730) based on the HTML document (720). The tree structure (730) may include a root node, at least one row node, at least one cell node, or at least one text node. According to one embodiment, a table (710) (or indicating a table (710)) in an HTML document (720).

[0146]

[0147] The tag) can be set as the top root node in the tree structure (730). According to one embodiment, two representing two rows in an HTML document (720) Tags can be set as row nodes or tr nodes that are child nodes of the root node in the tree structure (730). According to one embodiment, representing cells included in each row in an HTML document (720). Tags can be set as cell nodes or td nodes that are child nodes of each row node in the tree structure (730). Each tag may include a colspan indicating how many columns the cell extends to, or a rowspan indicating how many rows the cell extends to. For example, in an HTML document (720). It can indicate an extended cell corresponding to two columns and can be included as "td 2 colspan" in the tree structure (730).

[0148] According to one embodiment, text contained in each cell node in an HTML document (720) (e.g., Dog a , cat, Woof, Arf, Meow) can be set as text nodes, which are the lowest nodes in the tree structure (730). The text may include superscripts or subscripts. In an HTML document (720), superscripts may be indicated by tags, and subscripts are 태그로 지시될 수 있다. 예를 들어, 표(710)에 포함된 "Doga"는 HTML 문서(720) 및 트리 구조(730)에 "Doga"로 포함될 수 있다.

[0149] 도 8은 일 실시예에 따른 두 개의 표의 유사도를 측정하는 동작을 설명하기 위한 도면이다.

[0150] 도 8의 (a)는 도 7에 도시된 표(710) 및 트리 구조(730)(이하 각각 '제1 표(710)' 및 '제1 트리 구조(730)'라 칭함)를 나타내고 있고, 도 8의 (b)는 제2 표(810) 및 제2 트리 구조(830)를 나타내고 있다.

[0151] 일 실시예에 따르면, 제2 표(810)는 표의 일부를 인식한 결과에 대응하는 것일 수 있고, 제1 표(710)는 표의 전체를 인식한 결과에서 제2 표(810)에 대응되는 부분일 수 있다. 제2 트리 구조(830)는 제2 표(810)를 기반으로 생성된 것으로서, 도 7에서 설명된 방법과 유사한 방법으로 생성될 수 있다.

[0152] 일 실시예에 따르면, 제1 표(710)와 제2 표(810)의 유사도는 제1 트리 구조(730)와 제2 트리 구조(830)를 기반으로 측정될 수 있다. 예를 들어, 제1 표(710)와 제2 표(810)의 유사도는 TEDS에 기반한 다음 [수학식 4]를 기반으로 측정될 수 있다.

[0153]

[0154] [수학식 4]를 참조하면, 는 표 a(예: 제1 표(710))의 트리 구조(예: 제1 트리 구조(730))를 지시할 수 있고, 는 표 b(예: 제2 표(810))의 트리 구조(예: 제2 트리 구조(830))를 지시할 수 있다. 는 와 간의 트리 편집 거리(tree edit distance)(또는 정규화된 트리 편집 거리(normalized tree edit distance) 또는 OCR 편집 거리)를 나타낼 수 있다. 트리 편집 거리는 루트 노드를 제외한, 와 의 노드들과 텍스트들의 일치를 위한 편집 동작(예: 삽입, 삭제, 또는 대체 동작)을 수치화하여 표현한 것으로서, 트리 편집 거리가 작을수록 유사도가 높아질 수 있다. 는 에 포함된 노드 수를 지시할 수 있고, 는 에 포함된 노드 수를 지시할 수 있다. 는 및 중 최대값을 지시할 수 있다.

[0155] 일 실시예에 따르면, 도 8에 도시된 예에서, 제1 표(710)와 제2 표(810)의 편집 거리는 제1 트리 구조(730)에 포함된 노드들과 제2 트리 구조(830)에 포함된 노드들의 비교를 통해 결정될 수 있다. 비교 결과, 제1 트리 구조(730)와 제2 트리 구조(830)는 동일한 구조를 가지나 일부 상이한 텍스트를 포함할 수 있다. 예를 들어, 제1 트리 구조(730)에 포함된 cat(730) 및 Meow(734)는 제2 트리 구조(830)에 포함된 cap(832) 및 Me0w(834)와 상이할 수 있고, 이를 일치시키기 위해 제2 트리 구조(830)에 대한 편집 동작(예: 대체 동작)이 수행될 필요가 있다.

[0156] 일 실시예에 따르면, 제2 트리 구조(830)에 대한 편집 동작은 루트 노드를 제외한, tr 노드들로 구분되는 두 개의 노드 그룹 별로 수행될 수 있다. 두 개의 노드 그룹은 tr 노드 및 두 개의 td 노드를 포함하는 제1 노드 그룹과, tr 노드 및 세 개의 td 노드를 포함하는 제2 노드 그룹을 포함할 수 있다.

[0157] 제1 노드 그룹에 포함된 세 개의 노드 중 어느 하나의 td 노드에 포함된 cap(832)은 cat(730)으로 정정되어야 하므로, 트리 편집 거리는 총 노드 수(예: 3)와 편집 노드의 수(예: 1)에 기반한 로 결정될 수 있다.

[0158] 제2 노드 그룹에 포함된 네 개의 노드 중 어느 하나의 td 노드에 포함된Me0w(834)는 Meow(734)로 정정되어야 하므로, 트리 편집 거리는 총 노드 수(예: 4)와 편집 노드의 수(예: 1)에 기반한 로 결정될 수 있다.

[0159] 결과적으로, 에 대응하는 표(710)와 제2 표(810)의 트리 편집 거리는 로 결정될 수 있다.

[0160] 일 실시예에 따르면, 제1 트리 구조(730)와 제2 트리 구조(830)는 각각 일곱 개의 노드들을 포함하므로, 에 대응하는 최대 노드 수는 7로 결정될 수 있다.

[0161] 전술한 트리 편집 거리 와 최대 노드수 7을 기반으로, 제1 표(710)와 제2 표(810)의 유사도 TEDS는 다음 [수학식 5]에 나타난 바와 같이 결정될 수 있다.

[0162]

[0163] 일 실시예에 따르면, TEDS에 기반한 유사도는 설정된 범위 값(예: 0과 1사이의 값)으로 결정될 수 있고, 1에 가까울수록 유사도가 높음을 나타낼 수 있다.

[0164] 이하 도 9 내지 도 11을 참조하여, 전자 장치(100)의 동작을 상세히 설명한다. 일 실시예에 따르면, 도 9 내지 도 11에 도시된 동작들은 전자 장치(100)의 프로세서(130)에 의해 수행되는 것으로 이해될 수 있다. 도 9 내지 도 11 각각에 도시된 동작들은 도시된 순서에 국한되지 않고 다양한 순서로 수행될 수 있다. 일 실시예에 따르면, 도 9 내지 도 11 각각에 도시된 동작들 중 적어도 일부가 생략되거나, 도 9 내지 도 11 각각에 도시된 동작들보다 더 많은 동작들이 수행될 수도 있다.

[0165] 도 9는 일 실시예에 따른 전자 장치의 표 구조화 동작을 도시한 흐름도이다.

[0166] 도 9를 참조하면, 동작 902에서, 전자 장치(100)는 전자 문서(예: 도 2의 전자 문서(202)로부터 복수 개의 셀 박스 및 적어도 하나의 텍스트 박스를 식별할 수 있다. 일 실시예에 따르면, 전자 장치(100)는 표 인식을 위해 학습된 AI 모델(예: 도 2의 텍스트 인식 모델(210) 및 표 구조 인식 모델(220))을 기반으로, 복수 개의 셀 박스 및 적어도 하나의 텍스트 박스를 식별할 수 있다.

[0167] 일 실시예에 따르면, 복수 개의 셀 박스는 전자 문서 내의 표를 구성하는 복수 개의 셀의 경계를 식별하기 위한 박스 영역들을 나타낼 수 있다.

[0168] 일 실시예에 따르면, 적어도 하나의 텍스트 박스는 전자 문서의 표에 포함된 텍스트를 포함할 수 있고, 텍스트의 경계를 식별하기 위한 박스 영역을 나타낼 수 있다. 적어도 하나의 텍스트 박스에 포함된 텍스트는 단어, 띄어쓰기를 포함하는 문장, 또는 줄 바꿈을 포함하는 문단 중 적어도 하나를 포함할 수 있다.

[0169] 동작 904에서, 전자 장치(100)는 복수 개의 셀 박스 중 적어도 하나의 셀 박스에 적어도 하나의 텍스트 박스를 할당할 수 있다. 일 실시예에 따르면, 적어도 하나의 셀 박스에 할당된 적어도 하나의 텍스트 박스는 적어도 하나의 셀 박스에 가장 인접한 텍스트 박스를 포함할 수 있다. 복수 개의 셀 박스 중 적어도 하나의 또 다른 셀 박스는 설정된 거리 이내에 인접한 텍스트 박스가 존재하지 않을 경우, 어떠한 텍스트 박스도 포함하지 않을 수 있다.

[0170] 동작 906에서, 전자 장치(100)는 복수 개의 셀 박스의 좌표 정보에 포함된 복수 개의 X 좌표 및 복수 개의 Y 좌표를 기반으로, 적어도 하나의 경계선을 식별할 수 있다. 일 실시예에 따르면, 적어도 하나의 경계선은 표의 행 또는 열 중 적어도 하나를 구분하기 위한 경계선(예: 도 4c의 열 경계선(412) 및 / 또는 행 경계선(422))을 포함할 수 있다.

[0171] 동작 908에서, 전자 장치(100)는 적어도 하나의 경계선에 기반하여 복수 개의 셀 박스의 좌표 정보를 보정할 수 있다.

[0172] 동작 910에서, 전자 장치(100)는 복수 개의 셀 박스의 보정된 좌표 정보를 기반으로 복수 개의 셀 박스에 적어도 하나의 텍스트 박스를 재할당할 수 있다. 일 실시예에 따르면, 복수 개의 셀 박스에 적어도 하나의 텍스트 박스를 재할당하는 동작은, 복수 개의 셀 박스의 보정된 좌표 정보와 적어도 하나의 텍스트 박스의 좌표 정보를 기반으로, 복수 개의 셀 박스 각각에 가장 인접한 적어도 하나의 텍스트 박스를 할당하는 동작을 포함할 수 있다. 재할당 후 복수 개의 셀 박스 각각에는 텍스트 박스가 포함되거나 포함되지 않을 수 있고, 재할당 전과 후에 각 텍스트 박스가 포함되는 셀 박스는 동일하거나 상이할 수 있다.

[0173] 도 10은 일 실시예에 따른 전자 장치가 복수 개의 셀 박스의 좌표 정보를 업데이트하는 동작을 도시한 흐름도이다.

[0174] 일 실시예에 따르면, 도 10에 도시된 동작들은 도 9의 동작 908의 상세 동작들일 수 있다.

[0175] 도 10을 참조하면, 동작 1002에서, 전자 장치(100)는 적어도 하나의 경계선을 식별할 수 있다.

[0176] 동작 1004에서, 전자 장치(100)는 적어도 하나의 경계선에 열 경계선(예: 도 4c의 열 경계선(412))이 포함되어 있는지 여부를 판단할 수 있다. 일 실시예에 따르면, 열 경계선은 분류 알고리즘 또는 분류 모델(예: SVM)을 기반으로 표에 포함된 열들을 구분하기 위해 생성된 것일 수 있다.

[0177] 동작 1006에서, 전자 장치(100)는 적어도 하나의 경계선에 열 경계선이 포함되어 있는 것에 기반하여, 복수 개의 셀 박스와 연관된 복수 개의 X 좌표가 열 경계선을 기반으로 정렬되도록 복수 개의 X 좌표를 보정할 수 있다. 예를 들어, 전자 장치(100)는 열 경계선을 기준으로 이웃하고 있는 셀 박스들의 한 변이 열 경계선에 대응하도록 복수 개의 X 좌표를 보정할 수 있다.

[0178] 동작 1008에서, 전자 장치(100)는 적어도 하나의 경계선에 열 경계선이 포함되어 있지 않거나, 동작 1006을 수행한 것에 기반하여, 적어도 하나의 경계선에 행 경계선(예: 도 4c의 행 경계선(422))이 포함되어 있는지 여부를 판단할 수 있다. 일 실시예에 따르면, 행 경계선은 분류 알고리즘 또는 분류 모델(예: SVM)을 기반으로 표에 포함된 행들을 구분하기 위해 생성된 것일 수 있다.

[0179] 동작 1010에서, 전자 장치(100)는 적어도 하나의 경계선에 행 경계선이 포함되어 있는 것에 기반하여, 복수 개의 셀 박스와 연관된 복수 개의 Y 좌표가 행 경계선을 기반으로 정렬되도록 복수 개의 Y 좌표를 보정할 수 있다. 예를 들어, 전자 장치(100)는 행 경계선을 기준으로 이웃하고 있는 셀 박스들의 한 변이 행 경계선에 대응하도록 복수 개의 X 좌표를 보정할 수 있다.

[0180] 동작 1012에서, 전자 장치(100)는 복수 개의 보정된 X 좌표 및 / 또는 복수 개의 보정된 Y 좌표에 대한 정보가 포함되도록 복수 개의 셀 박스의 좌표 정보를 업데이트할 수 있다.

[0181] 도 11은 일 실시예에 따른 전자 장치의 신뢰도 제공 동작을 도시한 흐름도이다.

[0182] 일 실시예에 따르면, 도 11에 도시된 동작들은 도 9의 동작들 이후에 수행될 수 있다.

[0183] 도 11을 참조하면, 동작 1102에서, 전자 장치(100)는 복수 개의 셀 박스의 업데이트된 좌표 정보를 기반으로 표 업데이트를 수행할 수 있다. 일 실시예에 따르면, 전자 장치(100)는 복수 개의 셀 박스의 업데이트된 좌표 정보를 기반으로 격자 구조의 그리드 박스를 표시하고, 적어도 하나의 텍스트 박스를 복수 개의 셀 박스 중 가장 인접한 셀 박스에 재할당함으로써 표 업데이트를 수행할 수 있다.

[0184] 동작 1104에서, 전자 장치(100)는 업데이트된 표의 전체를 인식한 제1 결과와 업데이트된 표의 일부를 인식한 제2 결과를 획득할 수 있다.

[0185] 동작 1106에서, 전자 장치(100)는 제1 결과 내의 제2 결과에 대응하는 부분과 제2 결과의 유사도 정도에 기반하여 업데이트된 표에 대한 신뢰도를 결정할 수 있다. 일 실시예에 따르면, 전자 장치(100)는 업데이트된 표의 일부를 인식한 결과가 복수 개인 경우, 동작 1104와 동작 1106을 반복 수행할 수도 있다.

[0186] 동작 1108에서, 전자 장치(100)는 결정된 신뢰도에 대한 정보를 디스플레이(110)에 표시할 수 있다. 예를 들어, 전자 장치(100)는 알림창을 통해 신뢰도 정보를 표시하거나, 표 내의 신뢰도가 임계값 이하이거나 임계값을 초과하는 부분을 다양한 형태(예: 아이콘이나 메시지 표시, 또는 그래픽 형태)로 표시할 수 있다. 이에 기반하여, 사용자는 표 인식 결과를 사용할지 여부를 신뢰도에 기반하여 결정할 수 있는 편의를 가질 수 있다.

[0187] 본 개시에 따른 전자 장치는, 인스트럭션들을 저장하는 메모리; 및 상기 인스트럭션들을 실행하도록 구성되는 적어도 하나의 프로세서를 포함하며, 상기 인스트럭션들은, 상기 적어도 하나의 프로세서에 의하여 개별적으로 또는 집합적으로 실행될 때, 상기 전자 장치로 하여금: 전자 문서로부터 표에 포함되는 복수 개의 셀 박스 및 상기 표에 포함된 적어도 하나의 텍스트 박스를 식별하고, 상기 적어도 하나의 텍스 박스 각각은 텍스트를 포함하고; 상기 복수 개의 셀 박스의 제1 좌표 정보와 상기 적어도 하나의 텍스트 박스의 제2 좌표 정보에 기반하여, 상기 복수 개의 셀 박스 중 제1 대응되는 셀 박스에 상기 적어도 하나의 텍스트 박스 각각을 할당하고, 상기 복수 개의 셀 박스의 제1 좌표 정보에 포함된 복수 개의 X 좌표 및 복수 개의 Y 좌표를 기반으로, 상기 표의 행들 또는 열들 중 적어도 하나를 구분하기 위한 적어도 하나의 경계선을 식별하고, 상기 적어도 하나의 경계선에 기반하여 상기 복수 개의 셀 박스의 제1 좌표 정보를 보정하고, 상기 복수 개의 셀 박스의 보정된 제1 좌표 정보를 기반으로, 상기 복수 개의 셀 박스에 상기 적어도 하나의 텍스트 박스를 재할당할 수 있다.

[0188] 본 개시에 따른 전자 장치의 동작 방법은, 전자 문서로부터 표에 포함되는 복수 개의 셀 박스 및 상기 표에 포함된 적어도 하나의 텍스트 박스를 식별하는 동작, 상기 적어도 하나의 텍스트 박스 각각은 텍스트를 포함하고; 상기 복수 개의 셀 박스의 제1 좌표 정보와 상기 적어도 하나의 텍스트 박스의 제2 좌표 정보에 기반하여, 상기 복수 개의 셀 박스 중 제1 대응되는 셀 박스에 상기 적어도 하나의 텍스트 박스 각각을 할당하는 동작; 상기 복수 개의 셀 박스의 제1 좌표 정보에 포함된 복수 개의 X 좌표 및 복수 개의 Y 좌표를 기반으로, 상기 표의 행들 또는 열들 중 적어도 하나를 구분하기 위한 적어도 하나의 경계선을 식별하는 동작; 상기 적어도 하나의 경계선에 기반하여 상기 복수 개의 셀 박스의 제1 좌표 정보를 보정하는 동작; 및 상기 복수 개의 셀 박스의 보정된 제1 좌표 정보를 기반으로, 상기 복수 개의 셀 박스에 상기 적어도 하나의 텍스트 박스를 재할당하는 동작을 포함할 수 있다.

[0189] 본 문서의 다양한 실시예들 및 이에 사용된 용어들은 본 문서에 기재된 기술적 특징들을 특정한 실시예들로 한정하려는 것이 아니며, 해당 실시예의 다양한 변경, 균등물, 또는 대체물을 포함하는 것으로 이해되어야 한다. 도면의 설명과 관련하여, 유사한 또는 관련된 구성요소에 대해서는 유사한 참조 부호가 사용될 수 있다. 아이템에 대응하는 명사의 단수 형은 관련된 문맥상 명백하게 다르게 지시하지 않는 한, 상기 아이템 한 개 또는 복수 개를 포함할 수 있다. 본 문서에서, "A 또는 B", "A 및 B 중 적어도 하나", "A 또는 B 중 적어도 하나", "A, B 또는 C", "A, B 및 C 중 적어도 하나", 및 "A, B, 또는 C 중 적어도 하나"와 같은 문구들 각각은 그 문구들 중 해당하는 문구에 함께 나열된 항목들 중 어느 하나, 또는 그들의 모든 가능한 조합을 포함할 수 있다. "제 1", "제 2", 또는 "첫째" 또는 "둘째"와 같은 용어들은 단순히 해당 구성요소를 다른 해당 구성요소와 구분하기 위해 사용될 수 있으며, 해당 구성요소들을 다른 측면(예: 중요성 또는 순서)에서 한정하지 않는다. 어떤(예: 제 1) 구성요소가 다른(예: 제 2) 구성요소에, "기능적으로" 또는 "통신적으로"라는 용어와 함께 또는 이런 용어 없이, "커플드" 또는 "커넥티드"라고 언급된 경우, 그것은 상기 어떤 구성요소가 상기 다른 구성요소에 직접적으로(예: 유선으로), 무선으로, 또는 제 3 구성요소를 통하여 연결될 수 있다는 것을 의미한다.

[0190] 다양한 실시예들에 따르면, 상기 기술한 구성요소들의 각각의 구성요소(예: 모듈 또는 프로그램)는 단수 또는 복수의 개체를 포함할 수 있으며, 복수의 개체 중 일부는 다른 구성요소에 분리 배치될 수도 있다. 다양한 실시예들에 따르면, 전술한 해당 구성요소들 중 하나 이상의 구성요소들 또는 동작들이 생략되거나, 또는 하나 이상의 다른 구성요소들 또는 동작들이 추가될 수 있다. 대체적으로 또는 추가적으로, 복수의 구성요소들(예: 모듈 또는 프로그램)은 하나의 구성요소로 통합될 수 있다. 이런 경우, 통합된 구성요소는 상기 복수의 구성요소들 각각의 구성요소의 하나 이상의 기능들을 상기 통합 이전에 상기 복수의 구성요소들 중 해당 구성요소에 의해 수행되는 것과 동일 또는 유사하게 수행할 수 있다. 다양한 실시예들에 따르면, 모듈, 프로그램 또는 다른 구성요소에 의해 수행되는 동작들은 순차적으로, 병렬적으로, 반복적으로, 또는 휴리스틱하게 실행되거나, 상기 동작들 중 하나 이상이 다른 순서로 실행되거나, 생략되거나, 또는 하나 이상의 다른 동작들이 추가될 수 있다.

Claims

1. In an electronic device, Memory for storing instructions; and It includes at least one processor configured to execute the above instructions, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Identifying a plurality of cell boxes included in a table and at least one text box included in said table from an electronic document, wherein each of said at least one text box includes text; Based on the first coordinate information of the plurality of cell boxes and the second coordinate information of the at least one text box, each of the at least one text box is assigned to the first corresponding cell box among the plurality of cell boxes, and Based on a plurality of X coordinates and a plurality of Y coordinates included in the first coordinate information of the plurality of cell boxes, at least one boundary line for distinguishing at least one of the rows or columns of the table is identified, and Correcting the first coordinate information of the plurality of cell boxes based on the above at least one boundary line, and An electronic device that reallocates the at least one text box to the plurality of cell boxes based on the corrected first coordinate information of the plurality of cell boxes.

2. In Paragraph 1, The above at least one text box represents at least one box area for identifying the boundaries of each text included in the above at least one text box, and An electronic device in which the plurality of cell boxes represent box regions for identifying the boundaries of the plurality of cells included in the table.

3. In Paragraph 1 or 2, An electronic device wherein each text included in each of the above-mentioned at least one text box comprises at least one of one or more words, one or more sentences including at least one space, or one or more paragraphs including at least one line break.

4. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: One or more text boxes closest to the first corresponding cell box are assigned to the first corresponding cell box, The above one or more text boxes are an electronic device included in the above at least one text box.

5. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that obtains information about at least one boundary line using a pre-trained classification model based on the plurality of X coordinates and the plurality of Y coordinates.

6. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Based on the fact that the above at least one boundary line includes at least one first boundary line for separating the columns of the above table, the plurality of X coordinates are corrected so that the plurality of X coordinates are aligned based on the at least one first boundary line, and An electronic device that updates the coordinate information of the plurality of cell boxes to include information on the plurality of corrected X coordinates.

7. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Based on the fact that the above at least one boundary line includes at least one second boundary line for separating the rows of the above table, the plurality of Y coordinates are corrected so that the plurality of Y coordinates are aligned based on the at least one second boundary line, and An electronic device that updates the coordinate information of the plurality of cell boxes to include information on the plurality of corrected Y coordinates.

8. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Based on the fact that the above at least one boundary line includes at least one first boundary line for separating the columns of the above table, the plurality of X coordinates are corrected so that the plurality of X coordinates are aligned based on the at least one first boundary line, and Based on the fact that the above at least one boundary line includes at least one second boundary line for separating the rows of the above table, the plurality of Y coordinates are corrected so that the plurality of Y coordinates are aligned based on the at least one second boundary line, and An electronic device that updates the coordinate information of the plurality of cell boxes to include information on the plurality of corrected X coordinates and information on the plurality of corrected Y coordinates.

9. In Paragraph 1, Includes additional displays, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Update the table based on reassigning each of the at least one text box to the second corresponding cell box among the plurality of cell boxes, and A first result of recognizing the entire updated table and a second result of recognizing a part of the updated table are obtained, Based on the part within the first result corresponding to the second result and the degree of similarity between the second result and the first result, the reliability of the updated table is determined, and An electronic device that displays information regarding the above-determined reliability on the display.

10. In Paragraph 9, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that displays information on the display indicating that a part of the updated table has a reliability below the threshold value, based on the fact that the reliability determined above is below the threshold value.

11. In a method of operating an electronic device, An operation to identify a plurality of cell boxes included in a table from an electronic document and at least one text box included in the table, wherein each of the at least one text box includes text; An operation of assigning each of the at least one text box to a first corresponding cell box among the plurality of cell boxes based on the first coordinate information of the plurality of cell boxes and the second coordinate information of the at least one text box; An operation of identifying at least one boundary line for distinguishing at least one of the rows or columns of the table based on a plurality of X coordinates and a plurality of Y coordinates included in the first coordinate information of the plurality of cell boxes; An operation to correct the first coordinate information of the plurality of cell boxes based on the above at least one boundary line; and A method of operating an electronic device comprising the operation of reassigning at least one text box to the plurality of cell boxes based on corrected first coordinate information of the plurality of cell boxes.

12. In Paragraph 11, The above at least one text box represents a box area for identifying the boundaries of each text included in the above at least one text box, and A method of operation of an electronic device, wherein the plurality of cell boxes represent box regions for identifying the boundaries of the plurality of cells included in the table.

13. In Article 11 or Article 12, A method of operating an electronic device, wherein each text included in each of the above-mentioned at least one text box comprises at least one of one or more words, one or more sentences including at least one space, or one or more paragraphs including at least one line break.

14. In Paragraph 11, The operation of allocating at least one text box above is, The method includes the operation of assigning one or more text boxes closest to the first corresponding cell box to the first corresponding cell box, wherein The above one or more text boxes are included in the at least one text box, a method of operation of an electronic device.

15. In Paragraph 11, The operation of identifying at least one boundary line above is, An operation of obtaining information about at least one boundary line using a pre-trained classification model based on the plurality of X coordinates and the plurality of Y coordinates; and A method of operation of an electronic device comprising the operation of identifying at least one boundary line based on the information obtained above.

Images