Human-computer interaction method and apparatus, and electronic device

Abstract

Provided in the present application are a human-computer interaction method and apparatus and an electronic device. In the human-computer interaction method, in response to a text selection operation of a user, an electronic device can recognize text content displayed on a first interface and, on the basis of a recognition result, generate a tree structure of the text content, each node in the tree structure being a text block, and the text block of a lower-layer child node being located within a text block range of an upper-layer parent node. The user selects text in units of text blocks and, on the basis of the tree structure, the electronic device highlights in units of text blocks a text block selected by the user. When the user pastes the text, on the basis of the hierarchy of the tree structure, the electronic device displays on the second interface the content of the selected text block. On the basis of the constructed tree structure, the technical solution can take into account text selection efficiency, user selection intentions and post-pasting reading efficiency, thereby improving the interaction experience of text selection processes.

Description

Human-computer interaction method, device and electronic equipment

[0001] This application claims priority to the Chinese Patent Application No. 202411331279.1, filed on September 23, 2024, and entitled "Human-computer interaction method, device and electronic equipment", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of electronic devices, and more particularly, to a human-computer interaction method, device and electronic equipment. BACKGROUND

[0003] With the continuous development of electronic device technology, the functions of electronic devices are becoming more and more rich. In order to meet the user's needs for sharing and reusing text information, many electronic devices have text extraction functions to facilitate users to select and paste text information that cannot be directly edited in the interface. For example, the user can select text information by moving the cursor, wiping or clicking, etc. However, based on the current scheme, when the user uses the text extraction function, the operation efficiency is low, the selection intention cannot be met, or the reading after pasting is difficult, thereby resulting in poor interactive experience. SUMMARY

[0004] The present application provides a human-computer interaction method, device and electronic equipment, which can improve the interactive experience of the text extraction process.

[0005] In a first aspect, a human-computer interaction method is provided, comprising: displaying a first interface, the first interface comprising text content; in response to a first operation of a user, identifying the text content; generating a tree structure of the text content according to a result of identifying the text content, the tree structure being used to represent hierarchical information of the text content, wherein a parent node in the tree structure corresponds to a first text block, and a child node of the parent node corresponds to a second text block, the second text block being located in a region of the first text block; and in response to a second operation of the user, highlighting at least one second text block according to the tree structure.

[0006] In the present application, by constructing a tree structure of the text content of the first interface, the text content in the first interface can be hierarchized in the form of text blocks, thereby providing a correct reading order and supporting multiple reading orders. Accordingly, when extracting text, the order in which the user selects and pastes the text is more in line with general reading habits, thereby meeting the user's selection intention, improving reading efficiency and information dissemination integrity. Moreover, the user selects text based on text blocks, which is simple and efficient. Therefore, the scheme provided by the present application can balance the selection efficiency, the user's selection intention and the reading efficiency, thereby significantly improving the user's interactive experience.

[0007] In a possible implementation manner of the first aspect, according to the result of identifying the text content, the tree structure of the text content is generated, including: performing text block division on the text content according to first information, the first information including at least one of spatial information of the text, semantic information of the text, and visual saliency features in the first interface; determining a hierarchical relationship between the text blocks obtained by division according to second information, the second information including at least one of the spatial information of the text, the semantic information of the text, and the visual saliency features in the first interface; and performing sorting on the text blocks obtained by division according to third information, the third information including the spatial information of the text.

[0008] The tree structure is generated based on the first information, the second information, and the third information, which can achieve accurate segmentation of the original layout text and sorting in accordance with a general reading order. In the process of constructing the tree structure, compared with layout analysis that completely relies on image information, layout analysis based on text position information has stronger real-time performance.

[0009] In a possible implementation manner of the first aspect, performing text block division on the text content according to the first information includes: obtaining the first text block; determining at least one candidate division axis that passes through the first text block according to fourth information; determining a layout direction of the first text block according to the at least one candidate division axis, where the layout direction of the first text block is horizontal layout or vertical layout; and performing division on the first text block based on a first division axis in the at least one candidate division axis to obtain a plurality of second text blocks, where an extension direction of the first division axis is the same as the layout direction of the first text block.

[0010] Specifically, when the layout direction of the first text block is horizontal layout, the first division axis extends along the horizontal direction. When the layout direction of the first text block is vertical layout, the first division axis extends along the vertical direction.

[0011] The layout direction of the first text block is used to filter the division axis actually used for division from the at least one candidate division axis, which can eliminate unsuitable division axes and avoid unreasonable division manners.

[0012] In a possible implementation manner of the first aspect, the fourth information includes at least one of the following information: visual saliency features in the first interface, an alignment degree of the text in the horizontal direction, an alignment degree of the text in the vertical direction, a mapping relationship between the semantic information of the text and the spatial information of the text, and an arrangement density of the text.

[0013] There are many ways to obtain the candidate division axis, which can achieve accurate segmentation of the original layout text.

[0014] With reference to the first aspect, in a possible implementation manner, the alignment degree of the text in the landscape direction is determined according to at least one of a character size difference, an upper boundary difference, a lower boundary difference, and a vertical coordinate difference of a center point of a character in the text; and / or the alignment degree of the text in the portrait direction is determined according to at least one of a character size difference, a left boundary difference, a right boundary difference, and a horizontal coordinate difference of a center point of a character in the text.

[0015] With reference to the first aspect, in a possible implementation manner, the at least one candidate splitting axis includes one or more landscape candidate splitting axes and one or more portrait candidate splitting axes, and determining the layout direction of the first text block according to the at least one candidate splitting axis includes: performing pre-splitting on the first text block based on the one or more landscape candidate splitting axes to obtain a plurality of line text regions; determining line-to-line layout similarity of the first text block according to fifth information, the fifth information including at least one of an alignment manner difference between line text regions, a line width difference between line text regions, a text quantity difference between line text regions, a same-column text size difference between line text regions, and a line spacing difference between line text regions; performing pre-splitting on the first text block based on the one or more portrait candidate splitting axes to obtain a plurality of column text regions; determining column-to-column layout similarity of the first text block according to sixth information, the sixth information including at least one of an alignment manner difference between column text regions, a column width difference between column text regions, a text quantity difference between column text regions, a same-row text size difference between column text regions, and a column spacing difference between column text regions; and determining the layout direction of the first text block according to the line-to-line layout similarity and the column-to-column layout similarity, where in a case where the line-to-line layout similarity is greater than the column-to-column layout similarity, the layout direction of the first text block is landscape layout; and in a case where the column-to-column layout similarity is greater than the line-to-line layout similarity, the layout direction of the first text block is portrait layout.

[0016] The layout direction of the text block is determined by calculating the layout similarity, and the text block can be split in a manner that conforms to a general reading order.

[0017] With reference to the first aspect, in a possible implementation manner, before splitting the first text block based on a first splitting axis in the at least one candidate splitting axis, the method further includes: determining the first splitting axis from the at least one candidate splitting axis according to the layout direction of the first text block, where in a case where the layout direction of the first text block is landscape layout, the first splitting axis is the landscape candidate splitting axis; and in a case where the layout direction of the first text block is portrait layout, the first splitting axis is the portrait candidate splitting axis.

[0018] The layout direction of the first text block is used to filter the segmentation axis actually used for segmentation from the at least one candidate segmentation axis, so that inappropriate segmentation axes are eliminated, and the text block is segmented in a manner consistent with the general reading order.

[0019] In a possible implementation manner of the first aspect, the determining the at least one candidate segmentation axis according to the fourth information comprises: obtaining an adjacency matrix of the first text block, the adjacency matrix comprising distances of adjacent words in the first text block in the vertical direction or the horizontal direction; determining a position corresponding to a maximum value in the adjacency matrix in the first text block as the candidate segmentation axis; and the segmenting the first text block based on the first segmentation axis in the at least one candidate segmentation axis to obtain the plurality of second text blocks comprises: clustering words on both sides of the first segmentation axis to obtain the plurality of second text blocks, wherein distances of adjacent words in the second text blocks are less than the maximum value in the adjacency matrix.

[0020] The text block is divided based on the adjacency matrix, so that clusters of an arbitrary shape can be found, and the algorithm is not sensitive to abnormal points in the data set.

[0021] In a possible implementation manner of the first aspect, the method further comprises: determining a position corresponding to a second maximum value in the adjacency matrix in the second text block as a second segmentation axis, the second segmentation axis penetrating the second text block; and segmenting the second text block based on the second segmentation axis to obtain a plurality of third text blocks, wherein distances of adjacent words in the third text blocks are less than the second maximum value in the adjacency matrix.

[0022] Based on the adjacency matrix, all clusters can be found in one algorithm execution, and the number of clusters does not need to be specified in advance.

[0023] In a possible implementation manner of the first aspect, a ratio of the second maximum value to a vertical height of a smallest word in the first text block is greater than or equal to a preset threshold.

[0024] When a ratio of the distance between the adjacent words to the size of the smallest word in the first text block is less than the preset threshold, it can be considered that the adjacent words have a correlation relationship and are not suitable to be segmented into different text blocks.

[0025] In a possible implementation manner of the first aspect, the obtaining the first text block comprises: obtaining a horizontal alignment degree of each line of text and / or a vertical alignment degree of each column of text in the text content; and determining a plurality of continuous lines of text and / or a plurality of continuous columns of text as the first text block, wherein the horizontal alignment degree of the line of text is greater than a first threshold, and the vertical alignment degree of the column of text is greater than a second threshold.

[0026] With reference to the first aspect, in a possible implementation manner, the spatial information of the text includes a position of the text and a size of the text, and the position of the text includes a coordinate of the text in the first interface and / or a text line index of the text in the first interface.

[0027] The layout analysis is performed based on the spatial position of the text, and real-time performance is stronger.

[0028] With reference to the first aspect, in a possible implementation manner, the visual saliency feature includes at least one of the following: a text color, a text font, a color block, a shadow, and a frame line.

[0029] With reference to the first aspect, in a possible implementation manner, the sorting of the text blocks obtained by division according to the third information includes: in a case where the first text block is divided into a plurality of second text blocks based on a horizontal division axis, sorting the plurality of second text blocks from top to bottom according to the spatial information of the plurality of second text blocks; or in a case where the first text block is divided into a plurality of second text blocks based on a vertical division axis, sorting the plurality of second text blocks from left to right according to the spatial information of the plurality of second text blocks; and the sorting priority of the first text block is higher than the sorting priority of the second text block.

[0030] The direction of sorting the text blocks obtained by horizontal division from top to bottom is consistent with the layout direction of the divided text blocks, and conforms to the reading order of horizontal layout. The direction of sorting the text blocks obtained by vertical division from left to right is consistent with the layout direction of the divided text blocks, and conforms to the reading order of vertical layout.

[0031] With reference to the first aspect, in a possible implementation manner, before the text block division of the text content according to the first information, the method further includes: dividing all words in the text content into a plurality of text lines, each text line extending through the first interface in a horizontal direction; and sorting the plurality of text lines from top to bottom to obtain a text line index of each text line, which is used for text block division of the text content and / or sorting of the text blocks.

[0032] The text line index is a global line number of the text in the first interface, and can be used to determine the relative positions between the text blocks.

[0033] With reference to the first aspect, in a possible implementation manner, in response to a second operation of the user, the at least one second text block is highlighted according to the tree structure, including: determining a text block intended to be selected by the user according to a trajectory of the user selecting the text and the tree structure, wherein the trajectory of the user selecting the text passes through a region of the at least one second text block, and the text block intended to be selected by the user includes the at least one second text block; and highlighting the at least one second text block.

[0034] Based on the tree structure, the text block selected by the user according to the trajectory of the user selecting the text can be determined. When the user selects the text, the text selection can be efficiently and generally in line with the reading flow in units of text structures, without the problems of interleaved selection and out-of-order selection between different structures. When there are multiple possible arrangements between text blocks, the trajectory of the user selecting the text can freely express the text selection order in line with the user's intention.

[0035] With reference to the first aspect, in a possible implementation, according to the trajectory of the user selecting the text and the tree structure, the text block selected by the user according to the trajectory of the user selecting the text is determined, including: detecting that the trajectory of the user selecting the text switches from the region of the second text block to the region of the first text block, and marking the level and ordering information of the second text block selected in the first text block in the tree structure as a template; detecting that the trajectory of the user selecting the text switches from the region of the first text block to the region of the sibling node of the parent node, and determining, according to the template, that the text block selected by the user according to the trajectory of the user selecting the text includes the text blocks corresponding to the child nodes under the sibling node.

[0036] In this way, the text block selected by the user according to the trajectory of the user selecting the text can better meet the selection intention of the user and improve the selection efficiency.

[0037] With reference to the first aspect, in a possible implementation, the method further includes: highlighting the text blocks corresponding to the child nodes under the sibling node, where the visual saliency feature of the text blocks corresponding to the child nodes under the sibling node is different from the visual saliency feature of the second text block.

[0038] The visual saliency feature of the text block selected by mapping in the second interface is different from the visual saliency feature of the second text block in the second interface, which can prompt the user.

[0039] With reference to the first aspect, in a possible implementation, the second operation is any one of the following: an operation of selecting text based on a cursor, an operation of selecting text based on a brush, or an operation of selecting text based on a click.

[0040] With reference to the first aspect, in a possible implementation, the second operation is an operation of selecting text based on a click, and according to the trajectory of the user selecting the text and the tree structure, the text block selected by the user according to the trajectory of the user selecting the text is determined, including: determining, according to the position clicked by the user and the boundary information of the text blocks corresponding to the nodes in the tree structure, the text block of the lowest level corresponding to the position clicked by the user; and determining that the text block of the lowest level is the text block selected by the user according to the trajectory of the user selecting the text.

[0041] In this way, the user can select a row of normal text or an item of information with one click, improving the interaction efficiency.

[0042] With reference to the first aspect, in a possible implementation manner, the method further includes: in response to the paste operation of the user, displaying the content of the at least one second text block on the second interface according to the tree structure.

[0043] When pasting, the electronic device displays the at least one second text block on the second interface according to the hierarchy of the tree structure.

[0044] In the embodiments of the present application, the tree structure constructed for the text content can provide a general reading order to the user, so that when the user pastes the content of the selected text block on the second interface, the selected text block will also display the related information in the order consistent with the reading habit.

[0045] With reference to the first aspect, in a possible implementation manner, in response to the paste operation of the user, the content of the at least one second text block is displayed on the second interface according to the tree structure, including: displaying the content of the at least one second text block in the same row in the second interface, wherein the at least one second text block satisfies a first preset condition; the first preset condition includes at least one of the following: the at least one second text block has the same parent node; the at least one second text block has the same association relationship, and the association relationship is a subordinate relationship, a parallel relationship, or a start-end relationship.

[0046] The child nodes with the same parent node are generally close in position and are likely to appear in the small-range reading hot area of the user at the same time. Displaying the text blocks with the same parent node in the same row is consistent with the reading order of the user. The text blocks belonging to the same association relationship are close in reading order, and the text blocks belonging to the same association relationship are displayed together, which is consistent with the reading order and can improve the reading efficiency.

[0047] With reference to the first aspect, in a possible implementation manner, displaying the at least one second text block in the same row in the second interface includes: determining the display order of the content of the at least one second text block according to seventh information, and the seventh information includes at least one of the following information: spatial information of the at least one second text block in the first interface, semantic information of the at least one second text block, or at least one of the visual saliency features related to the at least one second text block in the first interface.

[0048] When different text blocks are displayed in the same row, the order of the text blocks can be rearranged, which can improve the reading efficiency.

[0049] With reference to the first aspect, in a possible implementation manner, in response to the paste operation of the user, the content of the at least one second text block is displayed on the second interface according to the tree structure, including: the content of the at least one second text block and the content of at least one fourth text block are displayed on the second interface, the content of the second text block and the content of the fourth text block are located in different rows of the second interface, and the second text block and the fourth text block satisfy a second preset condition; the second preset condition includes at least one of the following: the second text block and the fourth text block have different parent nodes; the second text block and the fourth text block have different association relationships, and the association relationship includes a subordinate relationship, a parallel relationship, and a start-end relationship.

[0050] Displaying the child nodes with different parent nodes or the child nodes with different association relationships in different rows can highlight the structural hierarchy between texts, and help improve information dissemination efficiency.

[0051] With reference to the first aspect, in a possible implementation manner, the content of the at least one second text block and the content of the at least one fourth text block are displayed on the second interface, including: the display order of the content of the at least one second text block and the content of the at least one fourth text block is determined according to eighth information, and the eighth information includes at least one of the following information: spatial information of the at least one second text block in the first interface, level information of the at least one second text block, spatial information of the at least one fourth text block in the first interface, or level information of the at least one fourth text block.

[0052] In the present application, the order is reorganized according to the position information and the level information of the text block, which helps restore the structural hierarchy and the association relationship contained in the original layout, and improves the reading efficiency.

[0053] With reference to the first aspect, in a possible implementation manner, the method further includes: displaying a format mark on the second interface, and the format mark is used to identify the association relationship between the at least one second text block and / or the level relationship of the at least one second text block in the tree structure.

[0054] Adding special symbols helps restore the structural hierarchy and the association relationship contained in the original layout, and improves the efficiency and integrity of information dissemination.

[0055] With reference to the first aspect, in a possible implementation manner, the format mark includes at least one of the following: parentheses, semicolons, dashes, serial numbers, bullets, line breaks, tabs, and vertical bar symbols.

[0056] With reference to the first aspect, in a possible implementation manner, the method further includes: detecting an operation of the user to format adjust the content of the second text block, and batch adjusting the display format of the text blocks at the same level as the second text block in the second interface.

[0057] The electronic device can provide a user with a function of batch adjusting display formats of text blocks in the same level, thereby improving format adjustment efficiency.

[0058] With reference to the first aspect, in a possible implementation, the granularity of the second text block is smaller than the granularity of the first text block.

[0059] With reference to the first aspect, in a possible implementation, among the plurality of child nodes of the parent node, adjacent child nodes are bidirectionally connected.

[0060] With reference to the first aspect, in a possible implementation, the leaf node in the tree structure corresponds to a word or a text line.

[0061] With reference to the first aspect, in a possible implementation, the first interface displays at least one of the following: a picture, a webpage, or a document.

[0062] With reference to the first aspect, in a possible implementation, the text content is not editable.

[0063] The second aspect provides a device, which is included in an electronic device, and has a function of implementing the behaviors involved in the first aspect and any possible implementation of the first aspect.

[0064] The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a display module or unit, an identification module or unit, a generation module or unit, a division module or unit, a sorting module or unit, an acquisition module or unit, a detection module or unit, a processing module or unit, and the like.

[0065] The third aspect provides an electronic device, including: one or more processors; one or more memories; the one or more memories store one or more computer programs, the one or more computer programs include instructions, when the instructions are executed by the one or more processors, the electronic device executes the method in the first aspect and any possible implementation of the first aspect.

[0066] The fourth aspect provides a computer-readable storage medium, including computer instructions, when the computer instructions run on an electronic device, the electronic device executes the method in the first aspect and any possible implementation of the first aspect.

[0067] The fifth aspect provides a computer program product including instructions, when the computer program product runs on a computer, the computer executes the method in the first aspect and any possible implementation of the first aspect.

[0068] In a sixth aspect, a chip is provided, which includes a processor and a data interface, the processor reads instructions stored on a memory through the data interface, and executes the method in the first aspect and any possible implementation of the first aspect.

[0069] Optionally, as an implementation form, the chip can further include a memory, the memory stores instructions, and the processor is configured to execute the instructions stored on the memory, and when the instructions are executed, the processor is configured to execute the method in the first aspect and any possible implementation of the first aspect.

[0070] The chip can be a field programmable gate array or an application specific integrated circuit.

[0071] The beneficial effects of the apparatus in the second aspect to the sixth aspect can refer to the beneficial effects of the method in the first aspect, which will not be described herein. BRIEF DESCRIPTION OF DRAWINGS

[0072] FIG. 1 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application.

[0073] FIG. 2 is a schematic diagram of a software structure of an electronic device according to an embodiment of the present application.

[0074] FIGS. 3-5 are schematic diagrams of an interface of a text extraction effect according to the prior art.

[0075] FIG. 6 is a schematic flowchart of a human-computer interaction method according to an embodiment of the present application.

[0076] FIG. 7 is a schematic diagram of a part of a tree structure according to an embodiment of the present application.

[0077] FIG. 8 is a schematic diagram of a part of a tree structure according to an embodiment of the present application.

[0078] FIG. 9 is a schematic flowchart of generating a tree structure in a human-computer interaction method according to an embodiment of the present application.

[0079] FIGS. 10-13 are schematic diagrams of determining a split axis position in a human-computer interaction method according to the present application.

[0080] FIG. 14 is a schematic diagram of a word feature according to an embodiment of the present application.

[0081] FIG. 15 is a schematic diagram of a text block layout direction according to an embodiment of the present application.

[0082] FIGS. 16-18 are schematic diagrams of determining a layout direction in a human-computer interaction method according to an embodiment of the present application.

[0083] FIGS. 19-21 are schematic diagrams of selecting a text in a human-computer interaction method according to an embodiment of the present application.

[0084] FIG. 22 is a partial schematic diagram of a tree structure according to an embodiment of the present application.

[0085] FIGS. 23-24 are schematic diagrams of selecting text in a human-computer interaction method according to an embodiment of the present application.

[0086] FIGS. 25-29 are schematic diagrams of pasting text in a human-computer interaction method according to an embodiment of the present application.

[0087] FIG. 30 is a schematic structural block diagram of an apparatus according to an embodiment of the present application.

[0088] FIG. 31 is a schematic structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0089] The technical solutions in the present application will be described below with reference to the drawings.

[0090] It should be noted that, in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" herein merely describes an association relationship of associated objects, and means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone.

[0091] In the embodiments of the present application, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second", and the like can explicitly or implicitly include one or more of the features. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two, "at least one" and "one or more" means one, two or more than two. The singular expression "one", "a kind", "the", "the above", "the" and "this" is intended to also include, for example, the expression "one or more", unless there is clear indication to the contrary in the context.

[0092] In the present specification, the reference to "one embodiment" or "some embodiments" and the like means that the specific features, structures or characteristics described in connection with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in further some embodiments" and the like appearing in different places in the present specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "include", "contain", "have" and their variants mean "including but not limited to", unless otherwise specifically emphasized.

[0093] For the convenience of understanding, some technical terms involved in the embodiments of the present application are explained and described below.

[0094] Tree structure: a hierarchical structure defined by branch relations, which is a finite set of n nodes. The basic unit of the tree structure is a node, and the link between nodes is called a branch, and the node and the branch form a tree.

[0095] Root node: the topmost node, which is the beginning of the tree structure.

[0096] Leaf node (or terminal node): the outermost node, which is the end of the tree structure. The leaf node has only a predecessor node and no successor node.

[0097] Parent node: the node of the previous layer (i.e. the predecessor node) of a certain node is called the parent node of the certain node.

[0098] Child node: the node of the next layer (i.e. the successor node) of a certain node is called the child node of the certain node.

[0099] Sibling node: nodes that have the same parent node are sibling nodes.

[0100] Ancestor node: all nodes on the branches from the root node to a certain node are called the ancestor nodes of the certain node.

[0101] Descendant node: any node in the subtree with a certain node as the root is called the descendant node of the certain node.

[0102] Cousin node: nodes whose parent nodes are in the same layer are cousin nodes.

[0103] Subtree: a tree with a child node of a certain node as the root, which is called the subtree of the certain node.

[0104] Level of a node: starting from the root node, the root node is the first layer, the children of the root are the second layer, and if a certain node is in the Lth layer, the root of its subtree is in the L+1th layer.

[0105] Parent-child relationship: in a hierarchical structure, the relationship between a node and its direct child node is defined as a parent-child relationship. There is a containment relationship between the parent node and its child node, and the parent node contains the information of its child node.

[0106] Sibling relationship: sibling nodes refer to all child nodes that have the same parent node. There is an equivalence relationship between these nodes, i.e. they share the same parent node but have no containment relationship with each other.

[0107] Superior-subordinate relationship: In a hierarchy, the superior-subordinate relationship refers to the relationship between a node and its indirect parent node. This relationship is often used to represent the connection between entities with a multi-level inclusion relationship.

[0108] Visual saliency feature: refers to the area or object in a visual scene that can attract people's attention. For real-world scenes, people will automatically identify the area of interest and process the area of interest, ignoring the area of no interest. The definition of visual saliency in computer vision is to simulate the human visual system's perception process of images using computer technology, extract the salient regions in the image (i.e. the area of human interest), so as to quickly search and locate the target of interest when facing natural scenes.

[0109] Density clustering algorithm (density-based spatial clustering of applications with noise, DBSCAN): defines a cluster as the maximum set of points connected by density, which can divide areas with high enough density into clusters and find clusters of arbitrary shape in a noisy spatial database.

[0110] Optical character recognition (OCR): refers to the technology of converting text in images or documents into editable and searchable data through scanning and analysis.

[0111] Layout: refers to the design, layout and organization of a page, including the position and distribution of text, pictures and other elements.

[0112] The method provided by the embodiments of the present application can be applied to electronic devices with display functions, for example, can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, notebook computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), smart home devices (such as smart screens), etc. The specific type of electronic device is not limited in the embodiments of the present application.

[0113] For example, FIG. 1 shows a schematic diagram of the hardware structure of an electronic device provided by the embodiments of the present application.

[0114] As shown in FIG. 1, the electronic device 100 can include a processor 110, a memory 120, a universal serial bus (USB) interface 130, a power supply 140, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a microphone 170C, a sensor module 180, a camera 191, a display 192, and the like.

[0115] The processor 110 can include one or more processing units. For example, the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), and the like. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0116] Among them, the controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0117] The memory in the processor 110 can also be provided for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that the processor 110 has just used or repeatedly uses. If the processor 110 needs to use the instructions or data again, it can be directly called from the memory, avoiding repeated access and reducing the waiting time of the processor 110, thus improving the efficiency of the system.

[0118] The NPU is a neural-network (NN) computing processor, which can quickly process input information by drawing on the structure of biological neural networks, such as the transmission mode between human brain neurons, and can also constantly self-learn. Through the NPU, intelligent cognitive applications of the electronic device 100 can be realized, such as image recognition, face recognition, speech recognition, text understanding, and the like.

[0119] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0120] For example, the processor 110 and the touch sensor 180B can communicate through an I2C bus interface to implement the touch function of the electronic device. The processor 110 and the camera 191 can communicate through a CSI interface to implement the shooting function of the electronic device. The processor 110 and the display screen 192 can communicate through a DSI interface to implement the display function of the electronic device.

[0121] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device. In some other embodiments of the present application, the electronic device can also use different interface connection methods or a combination of multiple interface connection methods in the above embodiments.

[0122] The power supply 140 is used to supply power to the processor 110, the memory 120, the display screen 192, the camera 191, the mobile communication module 150, and the wireless communication module 160, etc.

[0123] The wireless communication function of the electronic device 100 can be realized through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0124] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: the antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0125] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc. applied to the electronic device. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive an electromagnetic wave by the antenna 1, and perform filtering, amplification, etc. on the received electromagnetic wave, and transfer the processed electromagnetic wave to the modem processor to be demodulated. The mobile communication module 150 can also amplify a signal modulated by the modem processor, and radiate the amplified signal as an electromagnetic wave through the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be disposed in the same device as at least part of the modules of the processor 110.

[0126] The wireless communication module 160 can provide a solution for wireless communication including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. applied to the electronic device. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives an electromagnetic wave via the antenna 2, performs frequency modulation and filtering on the electromagnetic wave signal, and transmits the processed signal to the processor 110. The wireless communication module 160 can also receive a signal to be transmitted from the processor 110, perform frequency modulation and amplification on the signal, and radiate the processed signal as an electromagnetic wave through the antenna 2.

[0127] The electronic device 100 implements a display function through a GPU, a display screen 192, an application processor, etc.

[0128] The GPU is a microprocessor for image processing, and is connected to the display screen 192 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0129] The display screen 192 is configured to display images, videos, and the like. The display screen 192 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diodes (QLED), or the like. In some embodiments, the electronic device 100 can include one or N display screens 192, where N is a positive integer greater than 1.

[0130] The electronic device 100 can implement a photographing function through an ISP, the camera 191, a video codec, a GPU, the display screen 192, and an application processor, and the like.

[0131] The ISP is configured to process data fed back by the camera 191. The camera 191 is configured to capture still images or videos. An object generates an optical image through a lens and projects the optical image to a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. In some embodiments, the electronic device can include one or N cameras 191, where N is a positive integer greater than 1.

[0132] The video codec is configured to compress or decompress digital videos. The electronic device can support one or more video codecs. In this way, the electronic device can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, and the like.

[0133] The memory 120 is configured to store data and / or instructions.

[0134] The memory 120 can include an internal memory. The internal memory is used to store a computer-executable program code including instructions. The processor 110 performs various functional applications of the electronic device and data processing by executing the instructions stored in the internal memory. The internal memory can include a program storage area and a data storage area. The program storage area can store an operating system; the program storage area can also store one or more application programs (such as a gallery, contacts, etc.). The data storage area can store data created during use of the electronic device (such as images, contacts, etc.). In addition, the internal memory can include a high-speed random access memory, and can also include a non-volatile memory such as one or more disk storage devices, flash memory devices, universal flash storage (UFS), etc. In some embodiments, the processor 110 can cause the electronic device to perform the methods provided in the embodiments of the present application by executing the instructions stored in the internal memory and / or the instructions stored in the memory disposed in the processor 110.

[0135] The memory 120 can also include an external memory such as a Micro SD card to expand the storage capacity of the electronic device. The external memory can communicate with the processor 110 through an external memory interface to achieve data storage functions. For example, files such as music, videos, etc. are saved in the external memory.

[0136] The electronic device can achieve audio functions such as audio playback, recording, etc. through the audio module 170, the speaker 170A, the microphone 170C, the application processor, etc.

[0137] The sensor module 180 can include a pressure sensor 180A, a touch sensor 180B, a gyroscope sensor, an acceleration sensor, a distance sensor, and other sensors, etc.

[0138] The pressure sensor 180A is configured to sense a pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 192. The pressure sensor 180A can be of various types, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, etc. The capacitive pressure sensor can include at least two parallel plates of conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to the display screen 192, the electronic device detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device can also calculate the position of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with a touch operation intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view short messages is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold is applied to the short message application icon, an instruction to create a new short message is executed.

[0139] The touch sensor 180B, also referred to as a "touch panel". The touch sensor 180B can be disposed on the display screen 192, and the touch sensor 180B and the display screen 192 together form a touch screen, also referred to as a "touch panel". The touch sensor 180B is configured to detect a touch operation applied thereto or in the vicinity thereof. The touch sensor 180B can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 192. In other embodiments, the touch sensor 180B can also be disposed on the surface of the electronic device, which is different from the position of the display screen 192.

[0140] The gyroscope sensor can be used to determine the motion posture of the electronic device. For example, the gyroscope sensor can be used for shooting anti-shake, and can also be used for navigation, motion sensing game scenarios, etc. When the electronic device is stationary, the gyroscope sensor can detect the magnitude and direction of gravity.

[0141] The acceleration sensor can detect the acceleration of the electronic device in various directions (generally three axes).

[0142] The distance sensor is configured to measure distance. The electronic device can measure distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device can use the distance sensor to measure distance to achieve fast focusing.

[0143] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than illustrated, or combine certain components, or split certain components, or different arrangement of components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0144] The above introduces a possible hardware structure diagram of the electronic device. The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application take a layered architecture as an example to illustrate the software structure of the electronic device. It can be understood that the software system of the electronic device in the embodiments of the present application can also be other systems, such as a Harmony system, a Tizen system, a Sailfish system, etc., which will not be described one by one here.

[0145] FIG. 2 illustrates a software structure block diagram of an electronic device according to an embodiment of the present application. As shown in FIG. 2, the layered architecture divides the software into several layers, each of which has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, from top to bottom, an application layer, an application framework layer, a system runtime library layer, and a kernel layer.

[0146] The application layer can include a series of application packages (application). As shown in FIG. 2, the application package can include camera, gallery, calendar, call, map, weather, WLAN, Bluetooth, music, video, short message, voice assistant, etc. The application is mainly in the user interface (user interface, UI) aspect, and is usually written in JAVA language to call the interface of the application framework layer.

[0147] The application framework layer provides application programming interfaces (application programming interface, API) and programming frameworks for the applications of the application layer. The application framework layer includes some pre-defined functions. As shown in FIG. 2, the application framework layer can include a window manager, a content provider, a phone manager, a resource manager, a notification manager, a view system, etc.

[0148] The window manager is used to manage the window program. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, and intercept the screen, etc.

[0149] ​​​​The content provider stores and retrieves data and makes it accessible to applications. The data can include videos, images, audio, dialed and received calls, browsing history and bookmarks, phonebook, etc.

[0150] The telephony manager provides the communication functions of the electronic device. For example, management of call status (including call on, call off, etc.).

[0151] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0152] The notification manager enables applications to display notification information in the status bar, which can be used to convey messages of the notification type, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the completion of the download, message reminders, etc. The notification manager can also be a notification in the form of a chart or a scroll bar text appearing in the top status bar of the system, such as a notification of an application running in the background, or a notification in the form of a dialog window appearing on the screen. For example, prompting text information in the status bar, issuing a prompt sound, terminal device vibration, indicator light blinking, etc.

[0153] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0154] The system runtime library layer is a collection of a series of program libraries below the application framework layer, which can be divided into two parts, namely the system library and the application runtime (for example, the Android runtime).

[0155] The application runtime includes the core library and the virtual machine. The application runtime is responsible for the scheduling and management of the software system. The core library includes two parts: one part is the function function required by the java language to call, and the other part is the core library of the software system.

[0156] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform the management of the object life cycle, the management of the stack, the management of the thread, the management of the security and the exception, and the garbage collection, etc.

[0157] The system library is a support of the application framework, and can include multiple functional modules, such as a surface manager, media libraries, a two-dimensional graphics engine (such as SGL), a three-dimensional graphics processing library (such as OpenGL ES), an image processing library, and the like.

[0158] The surface manager is used to manage a display subsystem, and provides fusion of 2D and 3D layers for multiple applications.

[0159] The media libraries support playback and recording of multiple common audio, video formats, and static image files. The media libraries can support multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, and the like.

[0160] The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, composition, and layer processing, and the like.

[0161] The two-dimensional graphics engine is a drawing engine for 2D drawing.

[0162] The kernel layer is a layer between hardware and software, and is used to provide essential functions of an operating system, such as file management, memory management, process management, a network protocol stack, and the like. The kernel layer at least includes display drivers, camera drivers, audio drivers, sensor drivers, Bluetooth drivers, and the like.

[0163] For ease of understanding, the following embodiments of the present application will take an electronic device having the structure shown in FIGS. 1 and 2 as an example, and the man-machine interaction method provided by the embodiments of the present application will be specifically described in combination with the accompanying drawings and application scenarios.

[0164] With the continuous development of electronic device technology, the functions of electronic devices are becoming richer and richer. In order to meet the needs of users for sharing and reusing text information, many electronic devices have a text extraction function to facilitate users to select and paste text information that cannot be directly edited in an interface. For example, a user can select text information by moving a cursor, erasing, or clicking. However, based on the current scheme, the user experience is poor when using the text extraction function.

[0165] For example, in the text selection mode based on a cursor, multiple selection, missing selection, and wrong selection often occur. FIG. 3 shows an interface schematic diagram of text selection based on a cursor in the prior art.

[0166] As shown in (a) of FIG. 3, the content in the dashed box is the text (such as multiple itinerary information) intended to be selected by the user, but when the cursor is dragged to select the text, only part of the text is selected, and partial information is missing, resulting in less selection.

[0167] As shown in (b) of FIG. 3, the content in the dashed box is the text that the user intends to select, but when the cursor is dragged, the cursor jumps, for example, the adjusted (i.e., dragged) cursor jumps in position, or the unadjusted (i.e., not dragged) cursor jumps in position, such as the start cursor and the end cursor flipping (i.e., the order is reversed), making it difficult for the user to control, resulting in the selection of incorrect content.

[0168] As shown in (c) and (d) of FIG. 3, the content in the dashed box is the text that the user intends to select, but when the cursor is dragged to select the text in the dashed box, other text outside the dashed box is also selected, resulting in multiple selection.

[0169] The way of selecting text based on the cursor depends on the system's own ordering of the text. The above-mentioned missing selection and less selection mainly occur because the system's layout analysis of the page is wrong, resulting in disordered text, and the user cannot select the text according to the general reading order. The above-mentioned multiple selection occurs for two reasons. One reason is that the system's layout analysis of the page is wrong, resulting in disordered text, as shown in (c) of FIG. 3. The other reason is that the system's layout analysis result of the page only supports one reading order, so the user can only select the text in one order. For example, in (d) of FIG. 3, there are actually two potential reading orders for the trip section in the page, one is a horizontal reading order, paying attention to both the outbound trip and the return trip, and the other is a vertical reading order, paying attention to only the outbound trip or the return trip. If the system only supports the horizontal reading order, the user will select the return information when intending to select the content in the dashed box 1; if the system only supports the vertical reading order, the user will select the outbound information when intending to select the content in the dashed box 2.

[0170] In summary, the way of selecting text based on the cursor cannot well meet the user's selection intention, and the interactive experience is poor.

[0171] For example, in the way of selecting text based on smearing or clicking, the user needs to select or deselect the words one by one, which is low in operation efficiency. FIG. 4 shows an interface schematic diagram of the existing scheme of selecting text based on smearing or clicking.

[0172] As shown in (a) of FIG. 4, one scheme is that the system displays the text content in the page on a card separately, and the user clicks or smears the words one by one in any order. In this way, the card covers part of the area of the page, and the text layout in the card is inconsistent with the layout of the original page, so when selecting and adjusting the text, the user needs to screen the text by referring to the original picture, and needs to select or deselect one by one, which significantly affects the operation efficiency and increases the cognitive load of the interaction.

[0173] Referring to (b) of FIG. 4, another solution is that the system highlights the text box on the original page, and the user clicks or strokes the words in any order. In this way, the user needs to select or cancel accurately, and small-scale clicking or stroking increases the complexity of the operation, requiring the user to operate accurately and inefficiently.

[0174] In summary, the operation of selecting text by clicking or stroking is complex, inefficient, and has a poor interactive experience.

[0175] In addition, the process of pasting the selected text to another location also has a poor interactive experience. FIG. 5 shows the interface of the existing solution for text selection and pasting.

[0176] Referring to (a) of FIG. 5, taking the text extraction of travel information as an example, the associated text in the original page, such as start and end time, location content, etc., is scattered after pasting, and the seat level information is interleaved.

[0177] Referring to (b) of FIG. 5, taking the text extraction of weather information as an example, the text in the same line in the original page is cross-ordered after pasting, and the corresponding relationship is confusing.

[0178] The reason is that the display order of the text after pasting is consistent with the order of the system for recognizing the text. If the analysis of the system for the page is wrong, resulting in disordered text ordering, which does not conform to the general reading order, the pasted text is also difficult to reflect the reading order of the original page, reducing the reading efficiency and information dissemination integrity, and leading to a poor interactive experience.

[0179] In summary, the existing text extraction solution generally has a poor interactive experience. Therefore, the embodiments of the present application provide a human-computer interaction method, which can improve the interactive experience of text extraction for text content in a page.

[0180] FIG. 6 shows a schematic flowchart of a human-computer interaction method according to an embodiment of the present application. The method 200 shown in FIG. 6 is applied to an electronic device, which can have a hardware structure as shown in FIG. 1 and / or a software architecture as shown in FIG. 2. The method 200 includes steps S210 to S240.

[0181] S210, display a first interface including text content.

[0182] In the embodiments of the present application, the text content in the first interface cannot be directly edited. For example, the first interface displays a picture, a web page, or a document (such as a PDF document) with text content.

[0183] S220, in response to a first operation of a user, recognizing the text content.

[0184] For example, the first operation can be an operation of clicking a control for text recognition, or an operation of long-pressing a region where the text content is located. The first operation can trigger the word selection process.

[0185] In this step, the recognition result of the text content includes: words in the text content, and spatial information (e.g., position and size) of the words in the text content. The recognition result of the text content can be selected and pasted. The processing of the text content in the following steps of the present application is actually achieved by processing the recognition result of the text content.

[0186] It can be understood that step S220 is configured to convert the text content that cannot be directly edited into text content in an editable or searchable format (e.g., selectable and pasteable).

[0187] In some embodiments, the electronic device identifies the text content in the first interface based on an optical character recognition (OCR) technology. Accordingly, the recognition result of the text content is an OCR recognition result.

[0188] In some embodiments, in step S220, in response to the first operation of the user, the electronic device can further obtain visual saliency features in the first interface, such as text color, font, frame line, color block, shadow, etc.

[0189] S230, generating a tree structure of the text content according to the result of recognizing the text content, the tree structure being configured to represent hierarchical information of the text content, a parent node in the tree structure corresponding to a first text block, and child nodes of the parent node corresponding to second text blocks located in the region of the first text block.

[0190] In the present application, the text content is divided into a plurality of text blocks, and each text block corresponds to a node in the tree structure. The text block corresponding to the child node is located in the region of the text block corresponding to the parent node, and the content of the text block of the parent node includes the content of the text blocks of the child nodes.

[0191] In the present application, the parent node in the tree structure includes a plurality of child nodes. Accordingly, the first text block corresponding to the parent node includes a plurality of second text blocks.

[0192] S240, in response to the second operation of the user, highlighting at least one second text block according to the tree structure.

[0193] The second operation is a text selection operation. For example, the second operation can be an operation of selecting text based on a cursor, an operation of selecting text based on a brush, or an operation of selecting text based on a click. More specifically, the second operation can be a long-press operation, a sliding operation, or a click operation.

[0194] In some embodiments, the manner of highlighting the at least one second text block can include at least one of: highlighting, adding a border line, and adding an underline.

[0195] In the embodiments of the present application, by constructing the tree structure of the text content of the first interface, the text content in the first interface can be hierarchized in the manner of text blocks, so as to provide correct reading order and support multiple reading orders. Accordingly, when the text is extracted, the order of the text selected and pasted by the user is more in line with the general reading habit, so as to meet the selection intention of the user and improve the reading efficiency and information transmission integrity. Moreover, the user selects the text in the manner of text blocks, which is simple in operation and high in selection efficiency. Therefore, the scheme provided by the present application can take into account the selection efficiency, the selection intention of the user and the reading efficiency, so as to significantly improve the interactive experience of the user.

[0196] For ease of understanding, FIG. 7 and FIG. 8 respectively show a partial schematic diagram of the tree structure in combination with specific examples. It can be understood that FIG. 7 and FIG. 8 are only exemplary illustrations, which respectively show the hierarchical structure of a sub-tree in the tree structure.

[0197] Referring to FIG. 7, the text block a can be divided into text blocks b and c. In the tree structure, the text block a is the parent node of the text blocks b and c, and the text blocks b and c are sibling nodes. The text block b can be further divided into text blocks d and e, and the text block c can be further divided into text blocks g1, g2 and g3. In the tree structure, the text block b is the parent node of the text blocks d and e, and the text blocks d and e are sibling nodes. The text block c is the parent node of the text blocks g1, g2 and g3, and the text blocks g1, g2 and g3 are sibling nodes. The child nodes of the text block b and the child nodes of the text block c are cousins. The text block d can be further divided into text blocks f1, f2, f3, f4, f5 and f6. In the tree structure, the text block d is the parent node of the text blocks f1, f2, f3, f4, f5 and f6, and the text blocks f1, f2, f3, f4, f5 and f6 are sibling nodes. The text blocks e, f1, f2, f3, f4, f5, f6, g1, g2 and g3 only have the predecessor node and have no successor node, and thus are leaf nodes of the tree structure.

[0198] Referring to FIG. 8, the text block a can be divided into text blocks b and c. In the tree structure, the text block a is the parent node of the text blocks b and c, and the text blocks b and c are sibling nodes of each other. The text block c can be further divided into text blocks d1, d2, d3, d4 and d5, the text block c is the parent node of the text blocks d1, d2, d3, d4 and d5, and the text blocks d1, d2, d3, d4 and d5 are sibling nodes of each other. The text blocks d1, d2, d3, d4 and d5 can be further divided into smaller text blocks, for example, the text block d1 can be divided into text blocks e1, e2, e3, e4, e5 and e6, the text block d2 can be divided into text blocks f1, f2, f3, f4, f5 and f6, and the text block d5 can be divided into text blocks g1, g2, g3, g4, g5 and g6. The text blocks d1, d2, d3, d4 and d5 are the parent nodes of their child nodes, respectively. The child nodes of the text blocks d1, d2, d3, d4 and d5 are cousins of each other.

[0199] In the examples of FIG. 7 and FIG. 8, each parent node is a specific example of the first text block, and the child nodes under the parent node are specific examples of the second text block.

[0200] As can be seen from the drawings, the text blocks as child nodes are located in the text block area of their parent nodes, and the content of the text block of the parent node contains the content of the text block of the child node. In this way, based on the tree hierarchy, the user can select text in units of text blocks when selecting, and can select text blocks at low levels and high levels separately, thereby supporting more flexible text selection sequences. In addition, the text blocks have a hierarchical relationship, which is more in line with the general reading order.

[0201] The text blocks involved in the embodiments of the present application are text with a hierarchical structure, which can also be referred to as text structures.

[0202] The following describes the content of generating the tree structure in step S230 in more detail in combination with the drawings.

[0203] Referring to FIG. 9, in some embodiments, the above step S230 can specifically include steps S231 to S233.

[0204] S221, performing text block division on the text content according to first information.

[0205] The first information can include at least one of the following information: spatial information of the text, semantic information of the text or a visual saliency feature in the first interface.

[0206] In some embodiments, the spatial information of the text can include the position of the text and the size of the text. The position of the text can include the coordinates of the text in the first interface and / or the text line index of the text in the first interface.

[0207] For example, the coordinates of the text in the first interface can be corner coordinates, boundary coordinates, or center point coordinates.

[0208] For example, the text line index of the text in the first interface is obtained by dividing all the words of the text content into lines and then sorting from top to bottom. Through the text line index, the relative positions of two text blocks can be determined, for example, whether the two text blocks are located in the same text line in the first interface.

[0209] For example, the size of the text can include the vertical size and the horizontal size of the text.

[0210] It should be noted that the "text" involved in the present application can be a word, a phrase, a sentence, a paragraph, a chapter, or a text line.

[0211] In some embodiments, the semantic information of the text can include at least one of the following: content features (i.e., specific information contained in the text, such as topics, entities, etc.), semantic features (i.e., the meaning and context relationship of the text, such as word meaning, sentence meaning, etc.), structural features (i.e., the layout and organization of the text, such as paragraph division, title, etc.), functional features (i.e., the purpose and role of the text, such as informativeness, entertainment, etc.), temporal features (i.e., the time state of the event described by the text, such as past, present, future, etc.).

[0212] In some embodiments, the visual saliency feature in the first interface can include at least one of the following: text color, text font, color block, shadow, frame line, etc.

[0213] In the embodiments of the present application, when dividing the text blocks, the entire page is taken as the largest text block, the largest text block is first divided to obtain a plurality of small text blocks, and then each small text block is further divided to obtain smaller text blocks, until the smallest text block cannot be divided, thereby completing the division of the text blocks. Therefore, the divided text blocks are in a parallel relationship or a nested relationship.

[0214] In the embodiments of the present application, the first information is used to determine the division axis when dividing the larger text block. The following takes the first text block of the parent node and the second text block of the child node thereof as an example for illustration.

[0215] As an example, the first information includes a visual saliency feature in the first interface, such as color difference, shadow, frame line, color block, etc., and then the division axis can be determined according to the visual saliency feature.

[0216] For example, referring to (a) in FIG. 10, if there is a straight line in the region of the first text block B1 that passes through the first text block B1, the first text block B1 can be segmented with the straight line as the segmentation axis to obtain a plurality of second text blocks B2.

[0217] For example, referring to (b) or (c) in FIG. 10, if there are frame lines in the region of the first text block B1, the first text block B1 can be segmented with the position between adjacent frame lines as the segmentation axis to obtain a plurality of second text blocks B2.

[0218] For example, referring to (d) in FIG. 10, if there are color blocks in the region of the first text block B1, the first text block B1 can be segmented with the boundary of the color blocks as the segmentation axis to obtain a plurality of second text blocks B2.

[0219] As another example, the first information includes semantic information of the text, which can be used to determine the semantic relationship between the texts, and the segmentation axis can be determined according to the semantic relationship between the texts.

[0220] For example, referring to (c) in FIG. 10, the first text block B1 includes departure information and return information, and “departure” and “return” are semantically corresponding relationships at the same level, which implies that the two can be segmented. Therefore, the first text block B1 can be segmented with the position between the text related to the departure information and the text related to the return information as the segmentation axis to obtain a plurality of second text blocks B2.

[0221] For example, referring to (d) in FIG. 10, the left text in the first text block B1 all belong to time entities, which are obviously different from the right text in entity type, and semantically imply that they can be segmented. Therefore, the first text block B1 can be segmented with the position between the left time entity text and the right non-time entity text as the segmentation axis to obtain a plurality of second text blocks B2.

[0222] For example, referring to (a) in FIG. 11, the right text in the first text block B1 all belong to price entities, which are obviously different from the left text in entity type, and semantically imply that they can be segmented. Therefore, the first text block B1 can be segmented with the position between the right price entity text and the left price entity text as the segmentation axis to obtain a plurality of second text blocks B2.

[0223] For example, referring to (b) in FIG. 11, in the first text block B1, the text entity types of the two text regions at similar positions are consistent, presenting a high similarity semantic-spatial pattern, and therefore the first text block B1 can be segmented with the position between the two similar layouts as the segmentation axis to obtain a plurality of second text blocks B2.

[0224] As yet another example, the first information includes spatial information of the text, which can be used to determine a degree of alignment of the text in the landscape direction (or landscape alignment), a degree of alignment of the text in the portrait direction (or portrait alignment), or a density of arrangement of the text (e.g., a density of arrangement of the text in the landscape direction or a density of arrangement of the text in the portrait direction). The degree of alignment of the text in the landscape direction, the degree of alignment of the text in the portrait direction, or the density of arrangement of the text can be used to determine the split axis. The density of arrangement of the text can be determined according to a spacing between adjacent words in the text.

[0225] For example, referring to (a) and (b) in FIG. 12, there is a text region with a higher degree of alignment in the portrait direction (e.g., left alignment, right alignment, or center alignment, etc.) in the first text block B1, and the first text block B1 can be split using the boundary of the text region with the higher degree of alignment in the portrait direction as the split axis to obtain a plurality of second text blocks B2. As shown in (a) in FIG. 12, the text region to the left of the split axis is right-aligned in the portrait direction, and the right boundary of the text region can be used as the split axis. As shown in (b) in FIG. 12, the text region to the right of the split axis is left-aligned in the portrait direction, and the left boundary of the text region can be used as the split axis.

[0226] For example, referring to (c) in FIG. 12, there is a larger interval formed by two text regions in the first text block B1, and the position of the interval can be used as the split axis to split the first text block B1 to obtain a plurality of second text blocks B2.

[0227] For example, referring to (a) of FIG. 13, an adjacency matrix of the first text block B1 can be obtained first, which includes the distances between adjacent words in the first text block in the vertical direction (or includes the distances between adjacent words in the first text block in the horizontal direction). In some embodiments, an element in the adjacency matrix, also referred to as a spacing value, is a text spacing, for example, a line spacing (or a column spacing). In this embodiment, the smaller the spacing value, the greater the density, and the greater the spacing value, the smaller the density. By calculating the trend of the distances between adjacent words in the vertical direction (or the horizontal direction) in the adjacency matrix, values of significant jumps in the vertical distance (or the horizontal distance) can be found, which can be marked as density drop points (i.e., positions of increased text spacing). There can be multiple density drop points in the first text block B1, for example, 16, 38, and 108 as shown in the figure. When the first text block B1 is segmented, the density drop point with the maximum spacing value (such as 108) is used to perform a round of DBSCAN clustering on all the words in the first text block B1. The DBSCAN clustering will cluster the words with an adjacent density greater than the density corresponding to the density drop point together, and cluster the words with an adjacent density less than the density corresponding to the density drop point into different clusters, that is, cluster the words with a distance between adjacent words in the vertical direction (or the horizontal direction) less than the spacing value corresponding to the density drop point in one text block, and cluster the words with a distance between adjacent words in the vertical direction (or the horizontal direction) greater than the spacing value corresponding to the density drop point in different text blocks.

[0228] As shown in (b) of FIG. 13, based on the DBSCAN clustering, the first text block B1 can be divided into multiple second text blocks B2 according to the maximum value of the density drop point (such as 108), where the position of the density drop point is the position of the segmentation axis.

[0229] The DBSCAN clustering algorithm is an AI algorithm model that clusters points based on the distances between the points. Two important parameters in the model determine the performance of the clustering results: a distance matrix and a maximum neighbor distance value (eps). In this example, the distance matrix input into the algorithm model is the above-mentioned adjacency matrix of the words, and the eps is the value of the located density drop point.

[0230] As mentioned above, there can be multiple density drop points in the adjacency matrix, and after the first text block B1 is segmented into multiple second text blocks B2 through a round of DBSCAN clustering, the second text blocks B2 can also be subjected to DBSCAN clustering based on the second maximum value of the density drop point (for example, 38). For example, as shown in (c) of FIG. 13, based on the DBSCAN clustering, each second text block B2 can be divided into multiple third text blocks B3 according to the second maximum value of the density drop point (such as 38), where the position of the density drop point is the position of the segmentation axis.

[0231] That is, the adjacency matrix of the first text block B1 can determine the splitting axes of the multiple hierarchical text blocks for multiple rounds of splitting. Specifically, the adjacency matrix of the first text block B1 can have 0 to multiple density drop points, and for all the density drop points, DBSCAN clustering can be performed on all the words in the text block based on each density drop point in descending order of the interval value. In each round of DBSCAN clustering, the algorithm can cluster the words with an adjacency density greater than the density corresponding to the density drop point together to form a new text block. Thus, after each round of clustering, the original text block is divided into several smaller text blocks. With multiple rounds of DBSCAN clustering based on density drop points with smaller values, the text blocks obtained in the last iteration are divided into smaller text blocks until all the density drop points complete clustering.

[0232] In some embodiments, if the ratio of the value of a density drop point in the adjacency matrix to the size (e.g., the vertical size) of the smallest word in the first text block B1 is less than a preset threshold, the text block is no longer split based on the value of the density drop point. This is because the smaller the value of the density drop point, the higher the word arrangement density, and the smaller the distance between adjacent words. When the ratio of the distance between adjacent words to the size of the smallest word in the first text block B1 is less than the preset threshold, it can be considered that the adjacent words have a correlation and should not be split into different text blocks.

[0233] In some embodiments, if the distance between adjacent words in the first text block B1 is equal, there are 0 density drop points in the first text block B1, but a round of DBSCAN clustering can be performed on the first text block B1 based on the maximum value (all maximum values) in the adjacency matrix.

[0234] In the embodiments of the present application, in order to split a larger text block into smaller text blocks, the splitting axis determined according to the first information needs to pass through the text block to be split. For example, the splitting axis for splitting the first text block B1 into multiple second text blocks B2 needs to pass through the first text block B1.

[0235] In some embodiments, the degree of alignment of the text in the horizontal direction can be determined according to at least one of the size difference between words, the upper boundary difference between words, the lower boundary difference between words, and the horizontal coordinate difference between the center points of words in the text.

[0236] In some embodiments, the degree of alignment of the text in the vertical direction can be determined according to at least one of the size difference between words, the left boundary difference between words, the right boundary difference between words, and the vertical coordinate difference between the center points of words in the text.

[0237] For ease of understanding, referring to (a) of FIG. 14, a word has a frame capable of surrounding it, which is composed of an upper boundary, a lower boundary, a left boundary and a right boundary. The center point of the frame is the center point of the word. Referring to (b) of FIG. 14, the degree of alignment of text in the horizontal direction is used to describe the difference between the words in a row of text in the vertical direction, for example, the degree of horizontal alignment of text can be determined by the size difference between adjacent words in the vertical direction, the upper boundary difference, the lower boundary difference, the vertical coordinate difference of the center point, etc. Referring to (c) of FIG. 14, the degree of alignment of text in the vertical direction is used to describe the difference between the words in a column of text in the horizontal direction, for example, the degree of vertical alignment of text can be determined by the size difference between adjacent words in the horizontal direction, the left boundary difference, the right boundary difference, the horizontal coordinate difference of the center point, etc.

[0238] For example, the degree of alignment of text in the horizontal direction can be quantified by formula (1) as follows:

[0239] In formula (1):

[0240] Rx represents the degree of alignment of a row of text in the horizontal direction, and the larger Rx is, the higher the degree of horizontal alignment of the row of text is;

[0241] y i represents the vertical coordinate of the i-th word in a row of text;

[0242] y i-1 represents the vertical coordinate of the i-1-th word in a row of text;

[0243] h i represents the vertical size of the i-th word in a row of text;

[0244] h i-1 represents the vertical size of the i-1-th word in a row of text;

[0245] n represents the number of words included in a row of text.

[0246] In formula (1), the degree of alignment of a row of text in the horizontal direction can be calculated by the mean difference between adjacent words in the row of text.

[0247] For example, the degree of alignment of text in the vertical direction can be quantified by formula (2) as follows:

[0248] In formula (2):

[0249] Ry represents the degree of alignment of a column of text in the vertical direction, and the larger Ry is, the higher the degree of vertical alignment of the column of text is;

[0250] L ia horizontal coordinate representing a left border of the i-th word in a column of text;

[0251] L i-1 a horizontal coordinate representing a left border of the i-1-th word in a column of text;

[0252] R i a horizontal coordinate representing a right border of the i-th word in a column of text;

[0253] R i-1 a horizontal coordinate representing a right border of the i-1-th word in a column of text;

[0254] C i a horizontal coordinate representing a center point of the i-th word in a column of text;

[0255] C i-1 a horizontal coordinate representing a center point of the i-1-th word in a column of text;

[0256] h i a vertical dimension of the i-th word in a column of text;

[0257] h i-1 a vertical dimension of the i-1-th word in a column of text;

[0258] m represents a number of words included in a column of text.

[0259] In formula (2), the alignment degree of a column of text in the vertical direction can be calculated by means of the mean difference between adjacent words in the column of text.

[0260] In some embodiments, the splitting axis determined according to the first information can be used to determine the layout direction of the text block.

[0261] In the embodiments of the present application, the layout direction of the text block is consistent with the general reading order. A text block laid out in the horizontal direction is defined as a text block whose general reading order is to read from left to right along a line first, and then read from top to bottom along a column. A text block laid out in the vertical direction is defined as a text block whose general reading order is to read from top to bottom along a column first, and then read from left to right along a line. The text block laid out in the horizontal direction and the text block laid out in the vertical direction can be nested, i.e., a smaller text block laid out in the vertical direction can appear in the text block laid out in the horizontal direction. Similarly, a smaller text block laid out in the horizontal direction can appear in the text block laid out in the vertical direction. Of course, due to different splitting granularity, a smaller text block laid out in the horizontal direction can also appear in the text block laid out in the horizontal direction. Similarly, a smaller text block laid out in the vertical direction can appear in the text block laid out in the vertical direction.

[0262] For ease of understanding, the following takes the first text block of a parent node and the second text block of a child node thereof as an example for illustration.

[0263] Referring to (a) in FIG. 15, the general reading order of the first text block B1 in the solid line block is to read by line from left to right first and then read by line from top to bottom, and thus the layout direction of the first text block B1 in the solid line block is horizontal layout.

[0264] Referring to (b) in FIG. 15, the general reading order of the first text block B1 in the solid line block is to read by line from left to right first and then read by line from top to bottom, and thus the layout direction of the first text block B1 in the solid line block is horizontal layout. The general reading order of the second text block B2 is to read by column from top to bottom first and then read by column from left to right, and thus the layout direction of the second text block B2 is vertical layout. That is, the horizontal layout first text block B1 is nested with the vertical layout second text block B2.

[0265] Referring to (c) in FIG. 15, the general reading order of the first text block B1 in the solid line block is to read by column from top to bottom first and then read by column from left to right, and thus the layout direction of the first text block B1 in the solid line block is vertical layout.

[0266] Referring to (d) in FIG. 15, the general reading order of the first text block B1 in the solid line block is to read by column from top to bottom first and then read by column from left to right, and thus the layout direction of the first text block B1 in the solid line block is vertical layout. The general reading order of the second text block B2 is to read by line from left to right first and then read by line from top to bottom, and thus the layout direction of the second text block B2 is horizontal layout. That is, the vertical layout first text block B1 is nested with the horizontal layout second text block B2.

[0267] In some embodiments, if the splitting axes determined according to the first information are all horizontal splitting axes (i.e., extending in the horizontal direction and penetrating the first text block) or are all vertical splitting axes (i.e., extending in the vertical direction and penetrating the first text block), the first text block can be directly split.

[0268] For example, if the splitting axes determined according to the first information are all horizontal splitting axes, it can be considered that the layout direction of the first text block is horizontal layout, and thus the first text block can be split with horizontal splitting axes.

[0269] For another example, if the splitting axes determined according to the first information are all vertical splitting axes, it can be considered that the layout direction of the first text block is vertical layout, and thus the first text block can be split with vertical splitting axes.

[0270] In some embodiments, if the splitting axes determined according to the first information are both horizontal splitting axes or both vertical splitting axes, the layout direction of the first text block can be detected accurately to determine the splitting axis for actual splitting to avoid improper splitting leading to incorrect reading order. Of course, if the splitting axes determined according to the first information are both horizontal splitting axes or both vertical splitting axes, the layout direction of the first text block can also be detected accurately to filter out improper splitting axes to avoid improper splitting leading to incorrect reading order.

[0271] In some embodiments, taking splitting of the first text block as an example, step S231 can specifically include:

[0272] obtaining the first text block;

[0273] determining at least one candidate splitting axis according to fourth information, wherein the candidate splitting axis penetrates the first text block;

[0274] determining the layout direction of the first text block according to the at least one candidate splitting axis, wherein the layout direction of the first text block is horizontal layout or vertical layout;

[0275] splitting the first text block based on a first splitting axis in the at least one candidate splitting axis to obtain a plurality of second text blocks, and the first splitting axis extends in the same direction as the layout direction of the first text block.

[0276] That is, when splitting the first text block, at least one candidate splitting axis existing in the first text block is detected first, the layout direction of the first text block is determined according to the at least one candidate splitting axis, and then the first splitting axis for actual splitting is determined from the at least one candidate splitting axis based on the layout direction of the first text block to split the first text block. In this way, the split text blocks can be more consistent with the general reading order, and the sorting error of the text blocks can be avoided.

[0277] It should be noted that the above-mentioned extension direction of the first splitting axis is the same as the layout direction of the first text block, which can be understood as follows: if the layout direction of the first text block is horizontal layout, the first splitting axis extends horizontally; if the layout direction of the first text block is vertical layout, the first splitting axis extends vertically.

[0278] In some embodiments, the layout direction of the second text block is horizontal layout or vertical layout.

[0279] In this embodiment, the manner of determining the at least one candidate split axis according to the fourth information can be the same as the manner of determining the split axis according to the first information as described above, with the difference being that the candidate split axes determined here can not all be used to split the first text block, and the split axis actually used to split the first text block is selected from the at least one candidate split axis. For details, reference can be made to the related description of FIGS. 10 to 13, which will not be repeated here for brevity.

[0280] In some embodiments, the fourth information is the same as the first information. For example, the fourth information includes at least one of the following information: visual saliency features in the first interface (see FIG. 10), the alignment degree of the text in the landscape direction (see FIG. 12), the alignment degree of the text in the portrait direction (see FIG. 12), the mapping relationship between the semantic information of the text and the spatial information of the text (see FIG. 11), and the arrangement density of the text (see FIG. 13).

[0281] In some embodiments, the fourth information includes the arrangement density of the text, and the candidate split axis and the layout direction of the first text block can be determined by the arrangement density of the text. Specifically, the splitting process of the first text block in step S231 can be as follows:

[0282] Obtaining an adjacency matrix of the first text block, the adjacency matrix including distances of adjacent words in the first text block in the portrait direction or the landscape direction;

[0283] Determining a position corresponding to the maximum value in the adjacency matrix as a candidate split axis in the first text block;

[0284] Determining a layout direction of the first text block according to the at least one candidate split axis;

[0285] Determining a first split axis used to split the first text block according to the layout direction of the first text block;

[0286] Clustering words on both sides of the first split axis to obtain a plurality of second text blocks, wherein adjacent words in each second text block have a distance less than the maximum value in the adjacency matrix in the portrait direction or the landscape direction.

[0287] For details of splitting the first text block based on the adjacency matrix, reference can be made to the related description of FIG. 13, which will not be repeated here for brevity.

[0288] In some embodiments, step S231 can further include splitting the second text block, and the specific process can be as follows:

[0289] Determining a second split axis corresponding to the second maximum value in the adjacency matrix in the second text block, the second split axis penetrating the second text block;

[0290] The second text block is segmented based on the second segmentation axis, and a plurality of third text blocks are obtained, wherein the distance between adjacent words in a third text block in the longitudinal direction or the transverse direction is less than the second maximum value in the adjacency matrix.

[0291] For details about the multi-round segmentation based on the adjacency matrix, please refer to the related description of FIG. 13, which will not be repeated here for brevity.

[0292] In some embodiments, the ratio of the second maximum value to the longitudinal height of the smallest word in the first text block is greater than or equal to a preset threshold. For example, the preset threshold is greater than 0.

[0293] In some embodiments, while determining the at least one candidate segmentation axis, the candidate segmentation axis can be marked or positioned in the first text block.

[0294] The method for determining the layout direction according to the at least one candidate segmentation axis (or potential segmentation axis) provided in the present application will be described in detail below in combination with specific examples.

[0295] FIG. 16 shows a page diagram related to weather information, wherein (a) in FIG. 16 shows one or more longitudinal candidate segmentation axes detected for the first text block B1, and (b) in FIG. 16 shows one or more transverse candidate segmentation axes detected for the first text block B1.

[0296] FIG. 17 shows a page diagram related to travel information, wherein (a) in FIG. 17 shows one or more transverse candidate segmentation axes detected for the first text block B1, and (b) in FIG. 17 shows one or more longitudinal candidate segmentation axes detected for the first text block B1.

[0297] FIG. 18 shows a page diagram related to table-like layout, wherein (a) in FIG. 18 shows one or more transverse candidate segmentation axes detected for the first text block B1, and (b) in FIG. 18 shows one or more longitudinal candidate segmentation axes detected for the first text block B1.

[0298] It should be noted that the positions of the segmentation axes shown in the drawings of the embodiments of the present application are only schematic, and in addition, the area ranges of the text blocks are also only schematic, and should not be construed as a limitation of the present application.

[0299] In the embodiments of the present application, the process of determining the layout direction of the first text block according to the at least one candidate segmentation axis is as follows:

[0300] S1, pre-segmenting the first text block B1 based on the one or more transverse candidate segmentation axes detected, and obtaining a plurality of line text regions.

[0301] For example, as shown in (b) of FIG. 16, after the first text block B1 is pre-segmented based on the plurality of transverse candidate segmentation axes, line text regions C1, C2, C3, C4, C5, C6, C7 can be obtained. For example, as shown in (a) of FIG. 17 and (a) of FIG. 18, after the first text block B1 is pre-segmented based on the plurality of transverse candidate segmentation axes, line text regions C1, C2, C3, C4, C5, C6 can be obtained.

[0302] S2, determine line-to-line layout similarity of the first text block B1 according to the fifth information.

[0303] The fifth information can include at least one of alignment difference between line text regions, line width difference between line text regions, text quantity difference between line text regions, same column text size difference between line text regions, and line spacing difference between line text regions.

[0304] For example, the alignment of the line text region refers to the alignment of the text in the line text region, such as left alignment, right alignment, or center alignment. The line width of the line text region refers to the transverse dimension of the line text region. The text quantity of the line text region can refer to the number of words included in the line text region. The same column text between line text regions refers to the text corresponding in the vertical direction between different line text regions.

[0305] In the embodiments of the present application, according to the fifth information, the layout similarity between any two line text regions can be obtained. For example, in S2, the line-to-line layout similarity of the first text block B1 can be determined according to the layout similarity between adjacent line text regions.

[0306] For example, as shown in (b) of FIG. 16, the line-to-line layout similarity of the first text block B1 can be determined according to the layout similarity between the line text region C1 and C2, the layout similarity between the line text region C2 and C3, the layout similarity between the line text region C3 and C4, the layout similarity between the line text region C4 and C5, the layout similarity between the line text region C5 and C6, and the layout similarity between the line text region C6 and C7.

[0307] For example, if each line text region includes one line of text, the line-to-line layout similarity of the first text block B1 can be quantified by formula (3) as follows:

[0308] In formula (3),

[0309] h i,j represents the vertical dimension of the word in the i-th row and the j-th column;

[0310] h i,j-1 represents the vertical dimension of the word in the i-th row and the j-1-th column;

[0311] T i,j+1a vertical coordinate representing the upper boundary of the word in the i-th row and the j-th column;

[0312] B i,j a vertical coordinate representing the lower boundary of the word in the i-th row and the j-th column;

[0313] T i,j a vertical coordinate representing the upper boundary of the word in the i-th row and the j-th column;

[0314] B i,j-1 a vertical coordinate representing the lower boundary of the word in the i-th row and the j-th column;

[0315] m represents the number of rows;

[0316] n represents the number of columns.

[0317] In the formula (3), the interline layout similarity of the first text block B1 can be calculated by the mean of the difference between the adjacent row text areas, wherein the difference between the adjacent row text areas is obtained by calculating the difference between the same column texts in the adjacent row text areas.

[0318] For example, referring to (a) in FIG. 17, if each row text area includes multiple rows of texts, the interline layout similarity of the first text block B1 can be quantified by the formula (4) as follows:

[0319] In the formula (4):

[0320] h q,i,j represents the vertical dimension of the word in the i-th row and the j-th column in the q-th row text area;

[0321] h q,i,j-1 represents the vertical dimension of the word in the i-th row and the j-th column in the q-th row text area;

[0322] T q,i,j+1 a vertical coordinate representing the upper boundary of the word in the i-th row and the j-th column in the q-th row text area;

[0323] B q,i,j a vertical coordinate representing the lower boundary of the word in the i-th row and the j-th column in the q-th row text area;

[0324] T q,i,j a vertical coordinate representing the upper boundary of the word in the i-th row and the j-th column in the q-th row text area;

[0325] B q,i,j-1 a vertical coordinate representing the lower boundary of the word in the i-th row and the j-th column in the q-th row text area;

[0326] m represents the number of row text areas;

[0327] n represents the number of columns in a row text area (i.e., the number of columns in a row text area);

[0328] p represents the number of rows in a row text area (i.e., the number of rows in a row text area).

[0329] In formula (4), the inter-row layout similarity of the first text block B1 can be calculated by the mean of the differences between the adjacent row text areas, where the difference between the adjacent row text areas is obtained by calculating the difference between the corresponding row texts in the adjacent row text areas. The difference between the corresponding row texts in the adjacent row text areas includes the difference between the first row text of one row text area and the first row text of another row text area, the difference between the second row text of one row text area and the second row text of another row text area, and so on.

[0330] S3, pre-segmenting the first text block B1 based on the detected one or more longitudinal candidate segmentation axes to obtain a plurality of column text areas.

[0331] For example, as shown in (a) of FIG. 16, the column text areas P1, P2, P3, P4 and P5 can be obtained after pre-segmenting the first text block B1 based on the plurality of longitudinal candidate segmentation axes. For example, as shown in (b) of FIG. 17 or (b) of FIG. 18, the column text areas P1, P2, P3 and P4 can be obtained after pre-segmenting the first text block B1 based on the plurality of longitudinal candidate segmentation axes.

[0332] S4, determining the inter-column layout similarity of the first text block B1 according to sixth information.

[0333] The sixth information includes at least one of the alignment difference between the column text areas, the column width difference between the column text areas, the text number difference between the column text areas, the same row text size difference between the column text areas, and the column spacing difference between the column text areas.

[0334] For example, the alignment of the column text area refers to the alignment of the text in the column text area, such as top alignment, bottom alignment or middle alignment. The column width of the column text area refers to the horizontal dimension of the column text area. The text number of the column text area can refer to the number of words included in the column text area. The same row text between the column text areas refers to the corresponding text in the horizontal direction between different column text areas.

[0335] In the embodiments of the present application, according to the sixth information, the layout similarity between any two column text areas can be obtained. For example, in S4, the inter-column layout similarity of the first text block B1 can be determined according to the layout similarity between the adjacent column text areas.

[0336] For example, as shown in (a) of FIG. 16, the column-to-column layout similarity of the first text block B1 can be determined according to the layout similarity between the column text regions P1 and P2, the layout similarity between the column text regions P2 and P3, the layout similarity between the column text regions P3 and P4, and the layout similarity between the column text regions P4 and P5.

[0337] For example, if each column text region includes one column of text, the column-to-column layout similarity of the first text block B1 can be quantified by formula (5) as follows:

[0338] In formula (5):

[0339] h i,j represents the vertical dimension of the character in the ith row and the jth column;

[0340] h i,j-1 represents the vertical dimension of the character in the ith row and the j-1th column;

[0341] L i,j+1 represents the horizontal coordinate of the left boundary of the character in the ith row and the j+1th column;

[0342] R i,j represents the horizontal coordinate of the right boundary of the character in the ith row and the jth column;

[0343] L i,j represents the horizontal coordinate of the left boundary of the character in the ith row and the jth column;

[0344] R i,j-1 represents the horizontal coordinate of the right boundary of the character in the ith row and the j-1th column;

[0345] m represents the number of rows;

[0346] n represents the number of columns.

[0347] In formula (5), the column-to-column layout similarity of the first text block B1 can be calculated by the average of the differences between the adjacent column text regions, wherein the differences between the adjacent column text regions are obtained by calculating the differences between the same row texts in the adjacent column text regions.

[0348] For example, if each column text region includes multiple columns of text, the column-to-column layout similarity of the first text block B1 can be quantified by formula (6) as follows:

[0349] In formula (6):

[0350] h q,i,j represents the vertical dimension of the character in the ith row and the jth column in the qth column text region;

[0351] h q,i,j-1a vertical dimension of a word in the (i+1)th row and the jth column in the qth column text area;

[0352] L i,j+1 a horizontal coordinate of a left boundary of a word in the (i+1)th row and the jth column in the qth column text area;

[0353] R q,i,j a horizontal coordinate of a right boundary of a word in the (i+1)th row and the jth column in the qth column text area;

[0354] L q,i,j a horizontal coordinate of a left boundary of a word in the (i+1)th row and the jth column in the qth column text area;

[0355] R q,i,j-1 a horizontal coordinate of a right boundary of a word in the (i+1)th row and the jth column in the qth column text area;

[0356] m represents a number of rows in a column text area (i.e., a number of rows in a column text area);

[0357] n represents a number of column text areas;

[0358] k represents a number of columns in a column text area (i.e., a number of columns in a column text area).

[0359] In the formula (6), the column-to-column layout similarity of the first text block B1 is calculated by means of an average of differences between adjacent column text areas, wherein the differences between adjacent column text areas are obtained by calculating differences between corresponding column texts in adjacent column text areas. The differences between corresponding column texts in adjacent column text areas include a difference between a first column text of a column text area and a first column text of another column text area, a difference between a second column text of a column text area and a second column text of another column text area, and so on.

[0360] It should be noted that the aforementioned formula (1) to formula (6) are independent formulas, and the meanings of the parameters included in each formula are only defined for each formula respectively, and the parameter symbols and definitions in a certain formula do not limit other formulas.

[0361] S5, determines a layout direction of the first text block B1 according to the row-to-row layout similarity of the first text block B1 and the column-to-column layout similarity of the first text block B1.

[0362] Specifically, in a case where the row-to-row layout similarity of the first text block B1 is greater than the column-to-column layout similarity of the first text block B1, such as Sx is greater than Sy, it is determined that the layout direction of the first text block is horizontal layout. In a case where the column-to-column layout similarity of the first text block B1 is greater than the row-to-row layout similarity of the first text block B1, such as Sy is greater than Sx, it is determined that the layout direction of the first text block is vertical layout.

[0363] For example, according to (a) in FIG. 16, it can be seen that the differences between each column text region are small in terms of alignment, column width, number of texts, size of inline texts, and column spacing. According to (b) in FIG. 16, it can be seen that at least the line text regions C5 and C6 differ greatly from other line text regions in terms of alignment, size of inline texts, and line spacing. Thus, the column-to-column layout similarity of the first text block B1 is greater than the line-to-line layout similarity, and the first text block B1 is laid out in a portrait direction.

[0364] For example, according to (a) in FIG. 17, it can be seen that the differences between each line text region are small in terms of alignment, line width, number of texts, size of inline texts, and line spacing. According to (b) in FIG. 17, it can be seen that at least the column text region P4 differs greatly from other column text regions in terms of line width and number of texts. Thus, the line-to-line layout similarity of the first text block B1 is greater than the column-to-column layout similarity, and the first text block B1 is laid out in a landscape direction.

[0365] For example, according to (a) in FIG. 18, it can be seen that the differences between each line text region are small in terms of alignment, line width, number of texts, size of inline texts, and line spacing. According to (b) in FIG. 18, it can be seen that at least the column text regions differ greatly in terms of column width. Thus, the line-to-line layout similarity of the first text block B1 is greater than the column-to-column layout similarity, and the first text block B1 is laid out in a landscape direction.

[0366] In the above embodiments, the at least one candidate splitting axis includes one or more landscape candidate splitting axes and one or more portrait candidate splitting axes, and thus the line-to-line layout similarity and the column-to-column layout similarity of the first text block B1 can be obtained through the steps S1 to S5.

[0367] In some other embodiments, if the at least one candidate splitting axis is all landscape candidate splitting axes, the line-to-line layout similarity of the first text block can be obtained based on the steps S1 and S2, and when the line-to-line layout similarity of the first text block is greater than a first preset value, it is determined that the layout direction of the first text block is a landscape direction. Otherwise, it is considered that the first text block is not suitable for further splitting.

[0368] In some other embodiments, if the at least one candidate splitting axis is all portrait candidate splitting axes, the column-to-column layout similarity of the first text block can be obtained based on the steps S3 and S4, and when the column-to-column layout similarity of the first text block is greater than a second preset value, it is determined that the layout direction of the first text block is a portrait direction. Otherwise, it is considered that the first text block is not suitable for further splitting.

[0369] In some other embodiments, if no portrait splitting axis is detected through the first text block, it is determined that the layout direction of the first text block is a landscape direction by default.

[0370] In some embodiments, after determining the layout direction of the first text block, the electronic device can mark the layout direction of the first text block for use in sorting text blocks in the same level when constructing the tree structure.

[0371] In some embodiments, after determining the layout direction of the first text block B1, the first split axis can be determined from the at least one candidate split axis according to the layout direction of the first text block B1.

[0372] Specifically, in the case where the layout direction of the first text block B1 is horizontal layout, the first split axis is the aforementioned horizontal candidate split axis. In the case where the layout direction of the first text block B1 is vertical layout, the first split axis is the aforementioned vertical candidate split axis.

[0373] According to the screened split axis, the first text block can be split into smaller-granularity text blocks to achieve the columnar effect.

[0374] For example, referring to FIG. 16, the first text block B1 is vertically laid out, and thus the vertical candidate split axis is the split axis used to actually split the first text block B1, i.e., the first split axis. Based on the screened first split axis, the first text block B1 is split to obtain smaller-granularity second text blocks, such as the text blocks formed by the column text regions P1, P2, P3, and P4.

[0375] For example, referring to FIG. 17 and FIG. 18, the first text block B1 is horizontally laid out, and thus the horizontal candidate split axis is the split axis used to actually split the first text block B1, i.e., the first split axis. Based on the screened first split axis, the first text block B1 is split to obtain smaller-granularity second text blocks, such as the text blocks formed by the row text regions C1, C2, C3, C4, C5, and C6.

[0376] Screening the split axis actually used for splitting from the at least one candidate split axis according to the layout direction of the first text block can eliminate unsuitable split axes and avoid unreasonable splitting manners. For example, as shown in FIG. 18, the horizontal alignment layout of the table-like layout can find the candidate split axis formed by strict left-right alignment, and can also find a larger horizontal interval formed by the two sides of the text, but cannot simply split the table-like layout vertically according to the column, otherwise the general reading order is violated. Screening the candidate split axis according to the layout direction of the first text block can eliminate the column split axis in the table-like layout to ensure that the table-like layout is split according to the general reading order.

[0377] In some embodiments, for a layout of a table-like layout, the column division axis can also be removed in the following way. If it is determined that the column text regions on both sides of a vertical candidate division axis have a high degree of horizontal alignment (e.g., greater than a threshold A), and the size of the text in each column text region changes little (e.g., less than a threshold B), it can be determined that the vertical candidate division axis is suspected to be a column division axis of a table-like layout, and the vertical candidate division axis is removed when determining the first division axis.

[0378] In the above embodiments, the first division axis used to actually divide the first text block is selected from the candidate division axes based on the layout direction of the first text block after detecting the layout direction of the first text block. In some other embodiments, the process of determining the first division axis can be decoupled from the process of detecting the layout direction of the first text block. For example, the layout direction of the first text block can be determined by any of the layout direction detection methods described above. After determining the layout direction of the first text block, the electronic device can determine the first division axis based on the layout direction using any of the ways of determining the division axis based on the first information. For example, if it is determined that the layout direction of the first text block is horizontal layout, the horizontal division axis used to actually divide the first text block can be determined based on the first information. If it is determined that the layout direction of the first text block is vertical layout, the vertical division axis used to actually divide the first text block can be determined based on the first information.

[0379] For example, the electronic device can first detect whether there is a vertical candidate division axis that passes through the first text block. If not, it is directly determined that the layout direction of the first text block is horizontal layout. Based on the horizontal layout direction of the first text block, the electronic device can then determine the horizontal division axis used to divide the first text block based on the first information.

[0380] For another example, the electronic device can first detect whether there is a horizontal candidate division axis that passes through the first text block. If not, it is directly determined that the layout direction of the first text block is vertical layout. Based on the vertical layout direction of the first text block, the electronic device can then determine the vertical division axis used to divide the first text block based on the first information.

[0381] For yet another example, if the electronic device detects a horizontal candidate division axis and a vertical candidate division that pass through the first text block, and determines the layout direction of the first text block based on the horizontal candidate division axis and the vertical candidate division axis. Based on the layout direction of the first text block, the electronic device can then determine the division axis used to actually divide the first text block based on the first information.

[0382] In other words, both the process of detecting the layout direction of the first text block and the process of determining the actual division axis include the step of detecting the division axis, and the two methods of detecting the division axis can be the same or different.

[0383] In the embodiments of the present application, there are various ways to obtain the first text block.

[0384] As an example, the first text block can be obtained by segmenting a larger text block. The segmentation manner can refer to the segmentation manner of the first text block.

[0385] As another example, the degree of horizontal alignment of each line of text and / or the degree of vertical alignment of each column of text in the text content can be obtained; a plurality of continuous lines of text and / or a plurality of continuous columns of text are determined as the first text block, wherein the degree of horizontal alignment of the lines of text is greater than a first threshold value, and the degree of vertical alignment of the columns of text is greater than a second threshold value.

[0386] For example, a head-to-tail scan can be performed on the text content to find lines of text that are continuously horizontally aligned, or to find columns of text that are continuously vertically aligned, and the continuous lines of text or continuous columns of text are determined as the first text block. In some embodiments, the lines of text that are continuously horizontally aligned can be referred to as potential lines, and the columns of text that are continuously vertically aligned can be referred to as potential columns. A potential line is defined as a combination of words that are continuously top-aligned, bottom-aligned, or center-aligned. A potential column is defined as a combination of words that are continuously left-aligned, right-aligned, or center-aligned.

[0387] The above mainly describes the division of the text block in step S231 in detail. Next, referring back to FIG. 9, steps S232 and S233 are introduced.

[0388] As shown in FIG. 9, after the division of the text block is completed, in step S232, a hierarchical relationship between the divided text blocks is determined according to second information.

[0389] The second information can include at least one of spatial information of the text, semantic information of the text, and a visual saliency feature in the first interface. For specific descriptions of the second information, refer to the introduction of the first information above, and for brevity, no further description is given.

[0390] As an example, the second information can include spatial information of the text. For example, if it is determined according to the spatial information of a text block that the text block is located in the area of another text block, it can be determined that the level of the another text block is higher than the level of the text block. For another example, if it is determined according to the spatial information of two text blocks that the two text blocks are located in the same row of the first interface, it can be determined that the levels of the two text blocks are the same. For another example, if a text block is segmented to obtain smaller text blocks, the levels of the smaller text blocks are lower than the level of the segmented text block.

[0391] As another example, the second information can include semantic information of the text. For example, if the semantics of two text blocks are relative, such as “outbound” and “return”, it can be determined that the levels of the two text blocks are the same.

[0392] As a further example, the second information can include visual saliency features in the first interface, such as shading, color blocks, frame lines, etc. For example, if two text blocks are located in different color block areas or frame line areas, it can be determined that the two text blocks have the same level. As another example, if the color range of a text block is greater than that of another text block, it can be determined that the level of the text block is higher than that of the other text block.

[0393] In fact, after the foregoing step S231, the text content has been divided into a plurality of text blocks, which have a mutual containing relationship. In the tree structure constructed based on the divided text blocks, each text block is a node of the tree structure. Each layer of the tree structure represents a text structure, and the leaf nodes at the bottom layer are the original words identified in step S210.

[0394] After step S232, the hierarchicalization of the text content can be achieved.

[0395] S233, sorting the divided text blocks according to third information.

[0396] The third information includes spatial information of the text. Exemplarily, the spatial information of the text can include a position of the text and a size of the text. The position of the text can include coordinates of the text in the first interface and / or a text line index of the text in the first interface.

[0397] In this step, the positional relationship between two text blocks can be determined according to the spatial information of the text, so that the text blocks can be sorted in an order that is more in line with general reading habits.

[0398] In the embodiments of the present application, based on the hierarchical relationship between the text blocks determined in step S232, the text blocks can be sorted according to the following principles:

[0399] 1) The text blocks obtained by horizontal cutting are sorted from top to bottom;

[0400] 2) The text blocks obtained by vertical cutting are sorted from left to right;

[0401] 3) The upper structure is sorted first, and then the lower structure is sorted.

[0402] In principle 1), the positional relationship between the text blocks obtained by horizontal cutting can be determined according to the third information, for example, the relative positional relationship of the text blocks in the vertical direction can be determined according to the coordinates or text line index of the text blocks.

[0403] In principle 2), the positional relationship between the text blocks obtained by vertical cutting can be determined according to the third information, for example, the relative positional relationship of the text blocks in the horizontal direction can be determined according to the coordinates or text line index of the text blocks.

[0404] For example, referring to FIG. 7, the text block a (an example of the first text block) is divided into the text block b (an example of the second text block) and the text block c (an example of the second text block) by the horizontal division. Since the text block b is located above the text block c, the text block b is sorted in front and the text block c is sorted in back. For another example, referring to FIG. 8, the text block d1 (an example of the first text block) is divided into the text blocks e1, e2, e3, e4, e5 and e6 (examples of the second text block) by the horizontal division. According to the spatial information of the text blocks e1, e2, e3, e4, e5 and e6, the text blocks e1, e2, e3, e4, e5 and e6 are sorted from top to bottom.

[0405] It can be seen that the direction of sorting the text blocks obtained by the horizontal division from top to bottom is consistent with the layout direction of the divided text blocks, and conforms to the reading order of the horizontal layout.

[0406] For example, referring to FIG. 7, the text block c (an example of the first text block) is divided into the text blocks g1, g2 and g3 (examples of the second text block) by the vertical division. According to the spatial information of the text blocks g1, g2 and g3, the text blocks g1, g2 and g3 are sorted from left to right. For another example, referring to FIG. 8, the text block c (an example of the first text block) is divided into the text blocks d1, d2, d3, d4 and d5 (examples of the second text block) by the vertical division. According to the spatial information of the text blocks d1, d2, d3, d4 and d5, the text blocks d1, d2, d3, d4 and d5 are sorted from left to right.

[0407] It can be seen that the direction of sorting the text blocks obtained by the vertical division from left to right is consistent with the layout direction of the divided text blocks, and conforms to the reading order of the vertical layout.

[0408] It should be noted that the serial numbers on the branches shown in FIG. 7 or FIG. 8 represent the sorting order of the child nodes under each parent node.

[0409] Taking the first text block and the second text block as an example, the first text block has a higher sorting priority than the second text block, that is, the sorting of the first text block is prior to the sorting of the second text block. Referring to FIG. 8, the text block b is an upper structure of the text blocks d1 to d5, and the order of the upper structure is prior to the order of the lower structure. The text order arranged in this way is more in line with the general reading habit.

[0410] After the sorting is completed, the sorting of the text block at the lowest level is the final result. Referring to FIG. 7 or FIG. 8, if the text block a is the root node, the boxed numbers marked on the text block a represent the sorting order of each leaf node in the entire text block a, that is, the final word arrangement order and the default order of selecting the text from the first interface. As can be seen, the text order after rearrangement is more in line with the general reading habit.

[0411] The present application can realize accurate segmentation of the original page text and sorting in line with the general reading order by constructing a tree structure for the text content. In the process of constructing the tree structure, compared with the page analysis completely relying on image information, the page analysis based on text position information has stronger real-time performance.

[0412] In some embodiments, before step S231, step S230 can further include: dividing all words in the text content into a plurality of text lines, each text line extending through the first interface in the horizontal direction; and sorting the plurality of text lines from top to bottom to obtain a text line index of each text line. Here, the text line index is used for text block division of the text content and / or text block sorting. The text line index is global, and can also be referred to as a global line index.

[0413] For example, according to the text line index, it can be determined whether the words are in the same line or adjacent in the vertical direction, for use in text block division. For another example, according to the text line index, the relative positional relationship between the text blocks can be determined, for use in text block sorting.

[0414] By means of the foregoing steps S231 to S233, a tree structure can be constructed for the text content. Each layer of the tree structure represents a text block, and the leaf nodes at the bottom layer of the tree structure are single words or continuous lines. According to the tree structure, the user can select the text in the first interface in the unit of text block, and can flexibly select text blocks at different levels, which not only improves the selection efficiency, but also meets the selection intention of the user.

[0415] The content of the user selecting the text in step S240 will be described in more detail below with reference to the accompanying drawings.

[0416] In some embodiments, step 240 specifically comprises: determining a text block intended to be selected by the user according to the trajectory of the user selecting the text and the tree structure, wherein the trajectory of the user selecting the text passes through the region of at least one second text block, and the text block intended to be selected by the user comprises the at least one second text block; and highlighting the at least one second text block.

[0417] That is, based on the tree structure, the text block intended to be selected by the user can be determined according to the trajectory of the user selecting the text. The user can efficiently and in accordance with general reading flow select the text in units of text structures, without problems such as interleaved selection and out-of-order selection between different structures. When there are multiple possibilities for the arrangement order between text blocks, the trajectory of the cursor of the user selecting the text can freely express the selection order of the text in accordance with the user's intention.

[0418] FIG. 19 shows a schematic diagram of a page related to a trip, and for clarity, only the selected text blocks are shown in dashed border lines in the figure, and each closed dashed border line represents a text block.

[0419] Referring to FIG. 19, based on the constructed tree structure, the user can select the text based on the selection cursor (including the start position and the end position). Accordingly, the electronic device can determine the text block intended to be selected by the user according to the trajectory of the user dragging the cursor. The constructed tree structure and the start and end positions of the selection cursor support the user to select the text in the general reading order.

[0420] In one example, the user can select one piece of information about a trip ticket and multiple pieces of information about a return ticket in the first interface through the cursor. For example, in (a) of FIG. 19, the user can select one piece of information about a trip ticket by dragging the cursor. When the user drags the cursor to the right, as shown in (b) of FIG. 19, one piece of information about a return ticket can be continuously selected. When the user continues to drag the cursor downward, as shown in (c) of FIG. 19, a second piece of information about a return ticket can be continuously selected without selecting the second piece of information about a trip ticket.

[0421] In another example, the user can select multiple pieces of information about a trip or multiple pieces of information about a return in the first interface through the cursor. For example, in (a) of FIG. 19, the user can select one piece of information about a trip ticket by dragging the cursor. When the user drags the cursor downward, as shown in (c) of FIG. 19, one or more pieces of information about a trip ticket can be continuously selected without selecting the information about a return ticket.

[0422] When there are multiple possibilities for the arrangement order between structures, the trajectory of the cursor of the user selecting the text can freely express the selection order of the text in accordance with the user's intention. When the user wants to select multiple pieces of information about a trip ticket and the time and departure place, the user does not need to perform multiple selections, and the selection efficiency is high.

[0423] FIG. 20 shows a page diagram related to a trip, for clarity, only the selected text blocks are shown with dashed border lines, and the smaller text blocks within each selected text block are shown with solid border lines, each closed dashed border line and solid border line represents a text block.

[0424] Referring to FIG. 20, based on the constructed tree structure, the user can select text based on a smearing operation. Accordingly, the electronic device can determine the text block intended to be selected by the user according to the trajectory of the user's smearing.

[0425] For example, referring to (a) in FIG. 20, the user can smear horizontally to select one piece of information about a departure flight and one piece of information about a return flight.

[0426] For example, referring to (b) in FIG. 20, the user can smeer vertically to select multiple pieces of information about a departure flight. Or the user can smeer vertically to select multiple pieces of information about a return flight.

[0427] For example, referring to (c) in FIG. 20, the user can smeer horizontally first and then vertically to select one piece of information about a departure flight and multiple pieces of information about a return flight.

[0428] For example, referring to (d) in FIG. 20, the user can select multiple pieces of information about a departure flight and multiple pieces of information about a return flight through discontinuous smearing operations.

[0429] When there are multiple possible arrangements between the structures, the user's smearing path can express the hierarchical relationship of the text block intended to be selected. With respect to the smearing text line, fuzzy smearing is supported in units of structures to improve interaction efficiency and fault tolerance.

[0430] FIG. 21 shows a page diagram related to a transfer flow, for clarity, only the selected text blocks are shown with dashed border lines and solid border lines, and the upper text blocks of the selected text blocks are shown with dashed border lines and solid border lines, each closed dashed border line and solid border line represents a text block.

[0431] Referring to FIG. 21, based on the constructed tree structure, the user can select text based on a clicking operation. Accordingly, the electronic device can determine the text block intended to be selected by the user according to the trajectory of the user's clicking (including at least one clicking position).

[0432] For example, referring to (a) in FIG. 21, the user can click to select the text block of the lowest level node, that is, the text block of the leaf node of the tree structure, thereby selecting the balance information.

[0433] For example, referring to (b) in FIG. 21, the user can click to select the text block of the node of the next-to-lowest level (i.e., the second-to-last level), thereby selecting one piece of transfer record without date.

[0434] For example, referring to (c) in FIG. 21, the user can click the text block of the third-to-last node to select a transfer record with a date.

[0435] For example, referring to (d) in FIG. 21, the user can click the text block of the fourth-to-last node to select a transfer record of an entire month.

[0436] In the manner of selecting text based on the click operation, the user can select the text at the location by clicking the location, or select the lowest-level text block corresponding to the blank location by clicking the blank location. In this way, the user can select a line of normal text or an item of information with one click, improving the interaction efficiency.

[0437] In some embodiments, before the user clicks, the electronic device can display the outlines of the text blocks in the first interface to facilitate the user to determine the location of the text block to be selected.

[0438] In some embodiments, when the user clicks to select the text, the electronic device can display the outline of the lowest-level text block corresponding to the click location to facilitate the user to confirm whether it is the text block intended to be selected.

[0439] In some embodiments, the click location of the user selecting the text is used to determine the level of the text block selected by the user, to determine whether the user selects a single word, a low-level text block, or a larger range of high-level text blocks.

[0440] For example, when the user clicks to select the text, the electronic device can determine the lowest-level text block corresponding to the location clicked by the user according to the location clicked by the user and the boundary information of the text blocks corresponding to the nodes in the tree structure, and determine that the lowest-level text block is the text block intended to be selected by the user.

[0441] In the embodiments of the present application, the boundary of the text block at least encloses the text of the corresponding text block. In some embodiments, the boundary of the text block can also include a text-free area. The boundaries of the child nodes under the parent node do not overlap with each other. For example, the boundary of the text block can be a rectangular frame enclosing the corresponding text.

[0442] In some embodiments, referring to FIG. 22, when constructing the tree structure, an undirected graph can be constructed between the nodes at the same level in the tree structure, where the directly adjacent text blocks can be bidirectionally connected. The connection relationship between the nodes in the tree structure represents the possible paths of selecting the text, and the trajectory of the dragging cursor, the erasing, or the clicking represents the selection of the path. In this way, the tree structure can provide more selection orders for the user, and better meet the selection intention of the user.

[0443] Although the text is rearranged in a general reading order in constructing the tree structure, the user may choose to select the text in different orders for different information dissemination needs. The single order obtained by rearranging the text order cannot satisfy the user's multiple selection intentions. In this embodiment, a directed graph is constructed between nodes in the same layer to infer the user's intention to select the text block and the order between the selected text based on the user's selection trajectory.

[0444] As shown in (a) of FIG. 23, after the user selects the first word A in the first row of the text block Q1, if the user drags the selection cursor to the right, it is inferred that the user wants to continue to select the second word B to the right of the first word. If the user drags the selection cursor down after selecting the first word A, it is inferred that the user wants to continue to select the first word C in the second row below A. If the user continues to drag the cursor to the right after selecting A and C, it is inferred that the user wants to continue to select B and D (i.e., the first two rows of words in the text block Q1) to the right of A and C. When the user continues to drag the selection cursor to the right, it is inferred that the intended selection words have left the parent node of the current word and the user wants to continue to select the child nodes (words) under other parent nodes. Therefore, when it is detected that the user wants to continue to select words under other parent nodes in the same layer, the words selected by the user under the current parent node are recorded as a template, and the template (including the initial row order of the words in the text block) is used to quickly select the words under the new parent node.

[0445] In (c) of FIG. 23, the user has selected the first two rows of words in the text block Q1, and the first two rows of words of the parent node in the same layer are recorded as a selection template, which is used in the subsequent selection logic of other parent nodes in the same layer. Therefore, the user can directly select the first two rows of words in the text block Q2 to the right of the cursor, and continue to select the first two rows of words in the text block Q3 below the cursor, and so on.

[0446] As shown in the directed graph structure in FIG. 22, the implementation of this free selection method based on the user's selection trajectory relies on a directed graph structure formed based on the aforementioned constructed tree structure. In the directed graph, a bidirectional path is established between sibling nodes (i.e., sibling nodes) of the same parent node in the same layer in the tree structure. The bidirectional path represents that the user can select from a word corresponding to a child node to a word corresponding to a sibling node in any direction, or from a word under a parent node to a word under another parent node that is a sibling of the parent node.

[0447] Therefore, in some embodiments, in order to further improve the selection efficiency, the electronic device can also determine the text block selected by the user intention in the following manner: when it is detected that the selection area of the user switches from a child node (e.g., a leaf node) to a parent node, mark the selected child node at the current layer as a template; when it is detected that the selection area of the user switches to a sibling node of the parent node, the child nodes (e.g., leaf nodes) under the sibling node can be mapped and selected according to the template.

[0448] Taking the first text block and the second text block as an example, when the electronic device detects that the trajectory of the user selecting the text switches from the area of the second text block to the area of the first text block, the level and ordering information of the selected second text block in the tree structure in the first text block is marked as a template; when the electronic device detects that the trajectory of the user selecting the text switches from the area of the first text block to the area of the sibling node of the first text block, the text block selected by the user intention is determined according to the template to include the text blocks corresponding to the child nodes under the sibling node.

[0449] That is, the electronic device can map and select the text blocks corresponding to the child nodes of the sibling node of a parent node according to the selection of the user on the child nodes under the parent node, that is, map and select the cousin nodes of the selected child nodes. As an example, the level of the child node and its cousin node is the same, and the ordering or position of the child node under its parent node is the same as the ordering and position of the selected cousin node under its parent node. This will be further described below in combination with FIG. 23 and FIG. 24.

[0450] Referring to (a) in FIG. 23, the text blocks Q1, Q2, Q3, and Q4 are sibling nodes and have similar layouts, and each text block has multiple child nodes. As shown in (b) and (c) in FIG. 23, the user can select multiple child nodes under the text block Q1 by dragging the cursor. As shown in (d) in FIG. 23, when the trajectory of the user dragging the cursor switches from the area of the text block Q1 to the area of the text block Q2, the electronic device infers that the user intends to select the child nodes in the text block Q2 corresponding to the positions of the selected child nodes in the text block Q1. At this time, the electronic device automatically selects the corresponding text blocks in the text block Q2 without the user selecting them one by one. As shown in (e) in FIG. 23, when the trajectory of the user dragging the cursor switches from the area of the text block Q2 to the area of the text block Q3, the electronic device infers that the user intends to select the child nodes in the text block Q3 corresponding to the positions of the selected child nodes in the text block Q1. Similarly, at this time, the electronic device automatically selects the corresponding text blocks in the text block Q3 without the user selecting them one by one.

[0451] Referring to (a) of FIG. 24, the text blocks Q1 and Q2 are uncle nodes of each other, and each uncle node has a plurality of child nodes. As shown in (b) and (c) of FIG. 24, the user can select the plurality of child nodes under the text block Q1 (for example, a transfer record with a date) by a brushing operation. As shown in (d) of FIG. 24, when the brushing track of the user switches from the parent node area of the text block Q1 to the parent node area of the text block Q2, the electronic device infers that the user intends to select the child node in the text block Q2 corresponding to the position of the selected child node in the text block Q1. At this time, the electronic device automatically selects the corresponding text block (that is, another transfer record with a date) in the text block Q2 without the user selecting one by one.

[0452] In some embodiments, the electronic device highlights the mapped selected text block (that is, the text block corresponding to the child node under the brother node, that is, the uncle node of the child node).

[0453] In some embodiments, the visual saliency feature of the mapped selected text block is different from the visual saliency feature of the second text block. For example, the color background of the mapped selected text block is different from the background of the second text block; or the mapped selected text block has a flashing effect; or the mapped selected text block displays a structural outline.

[0454] In some embodiments, when the mapped selected text block is displayed in the second interface, the visual saliency feature of the mapped selected text block in the second interface can be different from the visual saliency feature of the second text block in the second interface to prompt the user.

[0455] In some embodiments, the second interface can include a selection control to facilitate the user to select to retain the content of the mapped selected text block or delete the content of the mapped selected text block.

[0456] The process of text selection is described in detail above in combination with the drawings. Referring back to FIG. 2, in some embodiments, the method 200 further includes:

[0457] S250, in response to the paste operation of the user, displaying the content of the at least one second text block in the second interface according to the tree structure.

[0458] In this step, the electronic device displays the content of the at least one second text block in the second interface according to the hierarchical information and the text ordering information provided by the tree structure.

[0459] It can be understood that in this step, when the user selects the text block of another parent node, the content of the selected text block of the user is also pasted and displayed in the second interface.

[0460] After the user finishes selecting the words in the reading order, the user often needs to paste the text in a formatted or plain text manner. In the embodiments of the present application, the tree structure constructed for the text content can provide the user with the general reading order, so that when the user pastes the selected text block content on the second interface, the selected text block will also display the relevant information in the order that conforms to the reading habit.

[0461] In one example, the text blocks pasted on the second interface are displayed in lines with the text blocks of the minimum level, i.e., each leaf node of the selected content is displayed in a line, and the contents of different leaf nodes are displayed in different lines. According to the ordering of the leaf nodes in the tree structure, the contents can be displayed on the second interface in the order that conforms to the reading order.

[0462] For example, referring to FIG. 25, the hierarchical structure of the text blocks shown in FIG. 25 is the same as that of the text blocks shown in FIG. 7. When the text block content shown in FIG. 25 is pasted and displayed on the second interface, the display form can be as shown in Paste Result 1, and all the leaf nodes are displayed in lines in the order in the hierarchical structure shown in FIG. 7.

[0463] For another example, referring to FIG. 26, according to the hierarchical structure of the text blocks shown in FIG. 26, when the text blocks are pasted and displayed on the second interface, the display form can be as shown in Paste Result 1, and all the leaf nodes are displayed in lines in the order in the hierarchical structure shown in FIG. 26.

[0464] For another example, referring to FIG. 27, the hierarchical structure of the text blocks shown in FIG. 27 is the same as that of the text blocks shown in FIG. 8. When the text block content shown in FIG. 27 is pasted and displayed on the second interface, the display form can be as shown in Paste Result 1, and all the leaf nodes are displayed in lines in the order in the hierarchical structure shown in FIG. 8.

[0465] In another example, when the text block content pasted on the second interface is displayed, the child nodes with the same parent node are displayed in the same line, and the child nodes with different parent nodes are displayed in different lines.

[0466] When reading, people generally read the texts with close positions together because these texts appear in the small-range reading hot area at the same time. Generally, the child nodes with the same parent node have a high degree of closeness in position, so the text blocks with the same parent node can be displayed in the same line. Here, the closeness in position of the texts can be determined by the granularity of the text blocks divided in the foregoing step. Generally, the smaller the granularity of the text blocks, the higher the closeness in position of the texts.

[0467] In some embodiments, the nodes belonging to the same low level (e.g., the level within the last three levels) under the same parent node are displayed in the same line.

[0468] In some embodiments, different text blocks are displayed in different lines.

[0469] For example, referring to FIG. 25 and FIG. 7, according to the hierarchy structure shown in FIG. 7, text blocks f1, f2, f3, f4, f5, f6 have the same parent node d, and thus text blocks f1, f2, f3, f4, f5, f6 can be displayed in the same line. Text blocks g1, g2, and g3 have the same parent node c, and thus text blocks g1, g2, and g3 can be displayed in the same line. According to FIG. 7, text blocks f1 / f2 / f3 / f4 / f5 / f6 / g1 / g2 / g3 have different parent nodes from text block e, and thus text block c is displayed in a separate line. Exemplarily, the display form of the text blocks shown in FIG. 7 in the second interface can be the pasting result 2 shown in FIG. 25.

[0470] For another example, referring to FIG. 26, according to the hierarchy structure shown in FIG. 26, text blocks d1, d2 have the same parent node b, and thus text blocks d1, d2 can be displayed in the same line. Text blocks e1, e2, e3, and e4 have the same parent node c, and thus text blocks e1, e2, e3, and e4 can be displayed in the same line. The parent nodes of text blocks d1, d2 are different from the parent nodes of text blocks e1, e2, e3, and e4, and thus text blocks d1, d2 are displayed in different lines from text blocks e1, e2, e3, and e4. Exemplarily, the display form of the text blocks shown in FIG. 26 in the second interface can be the pasting result 2 shown in FIG. 26.

[0471] For yet another example, referring to FIG. 27 and FIG. 8, according to the hierarchy structure shown in FIG. 8, text blocks e1, e2, e3, e4, e5, e6 have the same parent node d1, and thus text blocks e1, e2, e3, e4, e5, e6 can be displayed in the same line. Text blocks f1, f2, f3, f4, f5, f6 have the same parent node d2, and thus text blocks f1, f2, f3, f4, f5, f6 can be displayed in the same line. According to FIG. 8, text blocks d1 have different parent nodes from text blocks d2, and thus text blocks d1, d2 are displayed in different lines. Text block b belongs to a different hierarchy structure from text blocks d1, d2, and thus text block b is displayed in a separate line. Exemplarily, the display form of the selected text blocks shown in FIG. 27 in the second interface can be the pasting result 2 shown in FIG. 27.

[0472] In another example, when the text blocks pasted to the second interface are displayed, text blocks belonging to the same association relationship are displayed in the same line, and text blocks belonging to different association relationships or text blocks without association relationship are displayed in different lines.

[0473] People will actively search for the next text information with strong semantic correlation based on semantics when reading, for example, when reading the information on the ticket page, the departure time is found immediately to search for the arrival time, that is, the text reading order of strong semantic correlation is similar. When pasting, the texts belonging to the same correlation relationship are displayed together, which can improve the reading efficiency.

[0474] For example, with reference to FIG. 25 and FIG. 7, according to semantic analysis, text blocks f1, f2, f3, f4, f5, and f6 are information related to the itinerary, which specifically embodies the start-stop relationship, and thus the text blocks f1, f2, f3, f4, f5, and f6 can be displayed in the same row. Text blocks g1, g2, and g3 are all seat grade information, and thus the text blocks g1, g2, and g3 can be displayed in the same row. Text block e is information related to the price, which is different from the itinerary information and the seat grade information, and thus is displayed in a separate row. Exemplarily, the display form of the selected text blocks in the second interface shown in FIG. 25 can be the pasting result 2 shown in FIG. 25.

[0475] For another example, with reference to FIG. 26, according to semantic analysis, text blocks d1 and d2 are information related to the date, which specifically embodies the subordinate relationship, and thus the text blocks d1 and d2 can be displayed in the same row. Text blocks e1, e2, e3, and e4 are all information related to the transfer, and thus the text blocks e1, e2, e3, and e4 can be displayed in the same row. Text blocks b and c belong to different entities, and thus are displayed in separate rows. Exemplarily, the display form of the selected text blocks in the second interface shown in FIG. 26 can be the pasting result 2 shown in FIG. 26.

[0476] For another example, with reference to FIG. 27 and FIG. 8, according to the hierarchical structure shown in FIG. 8, text blocks e1, e2, e3, e4, e5, and e6 are information about the weather of the previous day, and thus the text blocks e1, e2, e3, e4, e5, and e6 can be displayed in the same row. Text blocks f1, f2, f3, f4, f5, and f6 are information about the weather of the current day, and thus the text blocks f1, f2, f3, f4, f5, and f6 can be displayed in the same row. According to FIG. 8, text block d1 and text block d2 are weather information of different dates, and thus the text blocks d1 and d2 are displayed in different rows. Text block b is title information, and thus text block b is displayed in a separate row. Exemplarily, the display form of the selected text blocks in the second interface shown in FIG. 27 can be the pasting result 2 shown in FIG. 27.

[0477] In some embodiments, when different text blocks are displayed in the same row, the text blocks can be arranged in sequence according to the sorting order of the text blocks in the tree structure (for example, according to the order from top to bottom or from left to right of the text blocks in the first interface), for example, as shown in the pasting result 2 shown in FIG. 25, the pasting result 2 shown in FIG. 26, and the pasting result 2 shown in FIG. 27.

[0478] In some embodiments, when different text blocks are displayed in the same row, the electronic device can rearrange the order of the text blocks. For example, the text blocks are arranged in the order of the font size of the text blocks from large to small. Referring to FIG. 26, the font sizes of the text blocks e1, e4, e3, e2 are successively reduced, and thus when e1, e2, e3, e4 are displayed in the same row, they are arranged in the order of e1, e4, e3, e2 successively, as shown in the paste result 4 in FIG. 26.

[0479] People usually pay attention to the text with larger font, bold, high color contrast, and unique font style when reading. The paste text is sorted based on such features, which can improve the reading efficiency.

[0480] In some embodiments, when different text blocks are displayed in different rows, the text blocks can be arranged in rows according to the reading order of the text blocks in the original layout or the hierarchy of the text blocks in the tree structure. For example, the structure with earlier reading order in the original layout has earlier row arrangement in the second interface, as shown in the paste result 2 in FIG. 25, the paste result 2 in FIG. 26, and the paste result 2 in FIG. 27. For example, the text blocks of the upper structure have earlier row arrangement in the second interface than the text blocks of the lower structure, for example, in the paste result 2 in FIG. 27, the text blocks of the higher-level “multi-day forecast” have earlier row arrangement than the text blocks of the weather information.

[0481] People usually read the text in the list type layout in an ‘F’ type mode, i.e., the text at the top and front of the list type layout is read first. The row arrangement according to the position information of the text lines in this application can improve the reading efficiency.

[0482] In summary, the rearrangement of the paste text in order and in rows helps to restore the structural hierarchy and the associated relationship in the original layout, and improves the efficiency and integrity of information dissemination.

[0483] In some embodiments, the selected content is displayed in the second interface with only one associated relationship in a row.

[0484] In some embodiments, when the selected text is pasted, the electronic device adds format markers to highlight the associated relationship between the text and / or the hierarchical relationship of the text blocks.

[0485] The addition of special symbols helps to restore the structural hierarchy and the associated relationship in the original layout, and improves the efficiency and integrity of information dissemination.

[0486] For example, the format markers include at least one of the following: parentheses, semicolons, dashes, serial numbers, bullets, line breaks, tabs, and vertical bar symbols.

[0487] For example, the parallel hierarchical relationship can be represented by adding line breaks.

[0488] For example, the cross-level relationship from the parent node to the child node can be represented by adding a line break and a tab. For example, the cross-level relationship between "multi-day forecast" and the weather information in the paste result 2 in FIG. 27. In some other embodiments, the tab can also be replaced by square brackets, bullets, numbers, etc.

[0489] For example, the association relationship between texts includes: subordinate relationship, from-to relationship (or start-end relationship), parallel relationship, and no relationship.

[0490] In some embodiments, the subordinate relationship can be defined between different level texts (e.g., different font sizes) that are close in the horizontal / vertical direction; or the subordinate relationship can be defined between different level texts (e.g., different font sizes) that are aligned in the horizontal / vertical direction.

[0491] For example, the subordinate relationship between texts can be highlighted by adding brackets, semicolons, etc. As an example but not limitation, the representation of the subordinate relationship can be like: A(B; C). Referring to the paste result 3 in FIG. 25, the subordinate relationships are: "06:10 (Shanghai Hongqiao Station)", "07:38 (Nanjing Station)". Referring to the paste result 3 in FIG. 26, the subordinate relationship is: "28 (Thursday)". Referring to the paste result 3 in FIG. 27, the subordinate relationships are: "Yesterday (07 / 02)", "Today (07 / 03)".

[0492] In some embodiments, the from-to relationship can be defined between same level texts connected by a dash or an arrow in the original layout.

[0493] For example, the from-to relationship between texts can be highlighted by adding a dash. As an example but not limitation, the representation of the from-to relationship can be like: A—B (association relationship supplementary text). Referring to the paste result 3 in FIG. 25, the from-to relationship is: "06:10 (Shanghai Hongqiao Station)—07:38 (Nanjing Station) (1 hour 28 minutes; G1970)". Referring to the paste result 3 in FIG. 27, the from-to relationships are: "33℃—28℃", "33℃—27℃".

[0494] In some embodiments, the parallel relationship can be defined between texts arranged equidistantly in the same line, or texts separated by the same symbol, or texts of the same size level.

[0495] For example, the parallel relationship between texts can be highlighted by adding a vertical bar symbol. As an example but not limitation, the representation of the parallel relationship can be like: A|B|C. Referring to the paste result 3 in FIG. 25, the parallel relationship is: "Second class with ticket|Second class with ticket|Business with ticket".

[0496] For example, the texts without direct relationship can be highlighted by adding semicolons. As an example but not limitation, the representation of the texts without relationship can be as follows: A; B; C. Referring to the pasting result 3 in FIG. 26, the texts without relationship are "accumulation fund; housing accumulation fund management center; balance 615.12; ￥3.90".

[0497] In the embodiments of the present application, when pasting the texts, the line aggregation (i.e., displaying in the same line), line sorting (displaying in different lines), adding special symbols and the like are helpful to restore the hierarchical structure, reading order and relationship. Therefore, when the selected texts are displayed in the second interface, the text structure is clear (e.g., block, segment, hierarchy), the reading order is reasonable (including reasonable reading order among the same hierarchical structure and reasonable containing order among different hierarchical structures), the relationship among the texts is highlighted, and the user can read the pasted texts in a consistent reading order, thereby improving the information transmission efficiency in text reading.

[0498] FIG. 28 shows a pasting result schematic diagram provided by the embodiments of the present application. As shown in FIG. 28, when pasting 1 to N hierarchical structures with relative semantics, the format markers can be added to reflect the corresponding relationship between the hierarchical structures. For example, the user selects one text information of the outbound journey and multiple text information of the return journey, and when pasting the texts, the one-to-many relationship between the return journey information and the outbound journey information can be reflected by adding bullets, numbers, or indentations and the like. For example, one outbound information is taken as a parent node, and multiple return information is taken as the child nodes of the outbound information.

[0499] In some embodiments, when it is detected that the user-selected texts have table headers or titles, even if the user does not select the corresponding table headers or titles, the "

title

[0500] In some embodiments, referring to (a) shown in FIG. 29, when the electronic device detects that the user adjusts the format of the text block in the second interface, the display format of the hierarchical text content can be automatically adjusted in batches, as shown in (b) of FIG. 29.

[0501] For example, the electronic device can adjust the display format of the hierarchical text content in batches, such as font, font size, color, shadow, indentation format, underline mark and the like.

[0502] In some embodiments, the selected text information in the present application can be pasted and displayed according to the following typesetting principles:

[0503] 1) Line aggregation principle: the texts of the same text block are displayed in the same line; the text blocks belonging to the same relationship are displayed in the same line.

[0504] 2) Subparagraph principle: the original page is displayed in different text blocks and unrelated text is sub-paragraphed; one line has only one paragraph relationship.

[0505] 3) Interline sorting principle: the text block on the top and left in the original page is sorted in the front; the upper text block is sorted in the front than the lower text block; and the special symbol such as tab can be added between the upper text block and the lower text block to indicate the containing relationship.

[0506] 4) Intra-line sorting principle: from large to small (font); from top to bottom (position); from left to right (position).

[0507] 5) Correlation highlighting principle: different symbols are used to highlight the correlation between texts, such as start-end relationship (A—B), parallel relationship (A|B|C), and subordinate relationship (A(B;C)).

[0508] The following takes the first text block and the second text block as an example to briefly describe the pasting method.

[0509] In some embodiments, step S250 can specifically include: displaying the content of at least one second text block in the same line in the second interface, wherein the at least one second text block meets a first preset condition.

[0510] The first preset condition includes at least one of the following:

[0511] The at least one second text block has the same parent node;

[0512] The at least one second text block has the same correlation relationship, which is a subordinate relationship, a parallel relationship, or a start-end relationship.

[0513] Referring to FIG. 7, the text blocks f1, f2, f3, f4, f5, and f6 are a specific example of the at least one second text block. Alternatively, the text blocks g1, g2, and g3 are a specific example of the at least one second text block.

[0514] Referring to FIG. 8, the text blocks f1, f2, f3, f4, f5, and f6 are a specific example of the at least one second text block. Alternatively, the text blocks e1, e2, e3, e4, e5, and e6 are a specific example of the at least one second text block.

[0515] Referring to FIG. 26, the text blocks d1 and d2 are a specific example of the at least one second text block. Alternatively, the text blocks e1, e2, e3, and e4 are a specific example of the at least one second text block.

[0516] In some embodiments, the second text block is within the last three layers in the tree structure.

[0517] In some embodiments, when displaying the content of the at least one second text block in the same row in the second interface, the display order of the content of the at least one second text block is determined according to seventh information. The seventh information includes at least one of the following: spatial information of the at least one second text block in the first interface, semantic information of the at least one second text block, or at least one of visual saliency features of the at least one second text block in the first interface. For the seventh information, refer to the aforementioned in-line ordering principles and the specific embodiments of the same-row display.

[0518] For example, a second text block located in the front and upper part of the first interface is displayed in the front in the second interface, and a second text block located in the back and lower part of the first interface is displayed in the back in the second interface.

[0519] For example, second text blocks that are semantically strongly related are displayed close to each other in the second interface.

[0520] For example, a second text block with special font, large font size, or special text color in the first interface is displayed in the front in the second interface.

[0521] In some embodiments, step S250 can specifically include: displaying the content of the at least one second text block and the content of the at least one fourth text block in the second interface, the content of the second text block and the content of the fourth text block being located in different rows in the second interface, the second text block and the fourth text block satisfying a second preset condition.

[0522] The second preset condition includes at least one of the following:

[0523] The second text block and the fourth text block have different parent nodes;

[0524] The second text block and the fourth text block have different association relationships, which include a subordinate relationship, a parallel relationship, or a start-end relationship.

[0525] Referring to FIG. 7, text blocks f1, f2, f3, f4, f5, and f6 are a specific example of the at least one second text block. Text blocks g1, g2, and g3 are a specific example of the at least one fourth text block.

[0526] Referring to FIG. 8, text blocks f1, f2, f3, f4, f5, and f6 are a specific example of the at least one second text block. Text blocks e1, e2, e3, e4, e5, and e6 are a specific example of the at least one fourth text block.

[0527] Referring to FIG. 26, text blocks d1 and d2 are a specific example of the at least one second text block. Text blocks e1, e2, e3, and e4 are a specific example of the at least one fourth text block.

[0528] In some embodiments, the second text blocks and the fourth text blocks are at more than four layers from the bottom in the tree structure.

[0529] In some embodiments, when the second interface displays the content of the second text blocks and the fourth text blocks in lines, the display order of the content of the at least one second text block and the content of the at least one fourth text block is determined according to eighth information. The eighth information includes at least one of the following information: the spatial information of the at least one second text block in the first interface, the hierarchical information of the at least one second text block, the spatial information of the at least one fourth text block in the first interface, or the hierarchical information of the at least one fourth text block. For the eighth information, please refer to the aforementioned line sorting principles and the specific embodiments of the line display.

[0530] For example, the position of the second text block in the first interface is higher than the position of the fourth text block in the first interface, and the display order of the second text block in the second interface is higher, and the display order of the fourth text block in the second interface is lower.

[0531] For example, the hierarchical level of the second text block is higher than the hierarchical level of the fourth text block, and the display order of the second text block in the second interface is higher, and the display order of the fourth text block in the second interface is lower.

[0532] In some embodiments, the method 200 can further include: displaying a format mark in the second interface, the format mark being used to identify the association relationship between the at least one second text block and / or the hierarchical relationship of the at least one second text block in the tree structure.

[0533] In some embodiments, the method 200 can further include: detecting an operation of the user to format adjust the content of the second text block, and batch adjusting the display format of the text blocks at the same hierarchical level as the second text block in the second interface.

[0534] For example, the format adjustment operation of the user to the content of the second text block includes at least one of the following: changing the text color, changing the text font, changing the text size, adding / removing the underline, adding / removing the serial number, adjusting the indentation format.

[0535] The human-computer interaction method provided by the embodiments of the present application is described in detail above in combination with FIGS. 1 to 29. The device embodiments of the present application will be described in detail below in combination with FIGS. 30 to 31. It should be understood that the description of the method embodiments corresponds to the description of the device embodiments, and therefore, the parts not described in detail can be referred to the foregoing method embodiments.

[0536] FIG. 30 shows a schematic structural diagram of an apparatus provided in an embodiment of the present application. The apparatus 300 can be located in the electronic device 100 shown in FIG. 1, or be a specific example of the electronic device 100. The apparatus 300 can perform each step in the method shown in FIG. 6, and can specifically implement the embodiments shown in FIGs. 7-29, and thus repeated description is omitted.

[0537] As shown in FIG. 30, the apparatus 300 can include a display unit 310, an identification unit 320, and a generation unit 330.

[0538] The display unit 310 can be configured to perform steps S210 and S240 and S250 in the method 200 shown in FIG. 6. The display unit 310 is configured to perform the steps related to display in the foregoing method embodiments, such as displaying a window, displaying an application interface, displaying a control, displaying a page, and the like.

[0539] The identification unit 320 can be configured to perform step S220 in the method 200 shown in FIG. 6. The identification unit 320 is mainly configured to perform the steps related to identifying text content in the foregoing method embodiments.

[0540] The generation unit 330 can be configured to perform step S230 in the method 200 shown in FIG. 6. The generation unit 330 is mainly configured to perform the steps related to generating a tree structure in the foregoing method embodiments. For example, the generation unit 330 is specifically configured to perform steps S231-S233 shown in FIG. 9.

[0541] Optionally, the apparatus 300 can further include a storage unit, configured to store program codes and data of the apparatus 300.

[0542] FIG. 31 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. The electronic device 400 shown in FIG. 31 can be a specific example of the electronic device 100 in FIG. 1.

[0543] The electronic device 400 shown in FIG. 30 includes a memory 410, a processor 420, and a bus 430. The memory 410 and the processor 420 are communicatively connected with each other through the bus 430.

[0544] The memory 410 can be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 410 can store programs, and when the programs stored in the memory 410 are executed by the processor 420, the processor 420 is configured to perform each step of the human-computer interaction method provided in an embodiment of the present application.

[0545] The processor 420 can be a general purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits, for executing programs to perform the human-computer interaction method of the embodiments of the present application.

[0546] The processor 420 can also be an integrated circuit chip having a processing capability of signals. In the implementation process, each step of the human-computer interaction method of the present application can be completed by the integrated logic circuit of hardware in the processor 420 or the instructions in the form of software. The processor 420 described above can also be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit, a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, which can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register, or other mature storage medium in the art. The storage medium is located in the storage 410, and the processor 420 reads the information in the storage 410 and executes the human-computer interaction method of the embodiments of the present application in combination with the hardware.

[0547] In some embodiments, the electronic device 400 further includes a communication interface 440. The communication interface 440 uses a transceiving device such as but not limited to a transceiver to realize the communication between the electronic device 400 and other devices or communication networks.

[0548] The bus 430 can include a path for transmitting information between various components (e.g., the storage 410, the processor 420, the communication interface 440) of the electronic device 400.

[0549] The application further provides an electronic device, comprising: one or more processors; one or more memories; the one or more memories store one or more computer programs, the one or more computer programs comprise instructions, when the instructions are executed by the one or more processors, the electronic device executes each step in the method shown in FIG. 6 or realizes the specific embodiments shown in FIGS. 7-29.

[0550] The application further provides a readable storage medium, comprising computer instructions, when the computer instructions are run on an electronic device, the electronic device executes each step in the method shown in FIG. 6 or realizes the specific embodiments shown in FIGS. 7-29.

[0551] The application further provides a chip, the chip stores instructions, when the instructions are run on an electronic device, the above human-computer interaction method can be realized.

[0552] The application further provides a computer program product, the computer program product stores programs or instructions, when the programs or instructions are run, the above human-computer interaction method can be realized.

[0553] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software mode depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0554] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0555] In several embodiments provided in the application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0556] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0557] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0558] The functions, if realized in the form of software functional units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0559] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A human-machine interaction method, characterized in that, The method comprises: displaying a first interface, the first interface comprising text content; in response to a first operation of a user, identifying the text content; generating a first structure of the text content according to a result of identifying the text content, the first structure being used to represent hierarchical information of the text content, wherein a first node in the first structure corresponds to a first text block, a second node associated with the first node corresponds to a second text block, and the second text block is located in a region of the first text block; in response to a second operation of the user, displaying at least one second text block in a first mode according to the first structure.

2. The method of claim 1, wherein, The generating of the first structure of the text content according to the result of identifying the text content comprises: performing text block division on the text content according to first information, the first information comprising at least one of spatial information of the text, semantic information of the text, and visual saliency features in the first interface; determining hierarchical relationships between the text blocks obtained through the division according to second information, the second information comprising at least one of the spatial information of the text, the semantic information of the text, and the visual saliency features in the first interface; performing sorting on the text blocks obtained through the division according to third information, the third information comprising the spatial information of the text.

3. The method of claim 2, wherein, The text block division on the text content according to the first information comprises: obtaining the first text block; determining at least one candidate division axis that penetrates the first text block according to fourth information; determining a layout direction of the first text block according to the at least one candidate division axis, wherein the layout direction of the first text block is horizontal layout or vertical layout; performing division on the first text block based on a first division axis in the at least one candidate division axis, and obtaining a plurality of second text blocks, wherein an extension direction of the first division axis is the same as the layout direction of the first text block.

4. The method of claim 3, wherein: the fourth information comprises at least one of the following information: visual saliency features in the first interface, an alignment degree of the text in the horizontal direction, an alignment degree of the text in the vertical direction, a mapping relationship between semantic information of the text and spatial information of the text, and a layout density of the text.

5. The method of claim 4, wherein: the alignment degree of the text in the horizontal direction is determined according to at least one of the following: a difference in size between characters in the text, a difference in upper edge between characters in the text, a difference in lower edge between characters in the text, and a difference in vertical coordinate of a center point between characters in the text; and / or the alignment degree of the text in the vertical direction is determined according to at least one of the following: a difference in size between characters in the text, a difference in left edge between characters in the text, a difference in right edge between characters in the text, and a difference in horizontal coordinate of a center point between characters in the text.

6. The method according to any one of claims 3 to 5, characterized in that, The at least one candidate division axis comprises one or more horizontal candidate division axes and one or more vertical candidate division axes, and the determination of the layout direction of the first text block according to the at least one candidate division axis comprises: performing pre-division on the first text block based on the one or more horizontal candidate division axes, and obtaining a plurality of line text regions; determine line layout similarity of the first text block according to fifth information, the fifth information comprising at least one of alignment difference of the line text intervals, line width difference of the line text intervals, text number difference of the line text intervals, same column text size difference of the line text intervals, line spacing difference of the line text intervals; perform pre-segmentation on the first text block based on one or more of the longitudinal candidate segmentation axes to obtain a plurality of column text intervals; determine column layout similarity of the first text block according to sixth information, the sixth information comprising at least one of alignment difference of the column text intervals, column width difference of the column text intervals, text number difference of the column text intervals, same line text size difference of the column text intervals, column spacing difference of the column text intervals; determine layout direction of the first text block according to the line layout similarity and the column layout similarity, wherein, in a case where the line layout similarity is greater than the column layout similarity, the layout direction of the first text block is horizontal layout; in a case where the column layout similarity is greater than the line layout similarity, the layout direction of the first text block is vertical layout.

7. The method of claim 6, wherein, before segmenting the first text block based on a first segmentation axis of the at least one candidate segmentation axis, the method further comprises: determining the first segmentation axis from the at least one candidate segmentation axis according to the layout direction of the first text block, wherein, in a case where the layout direction of the first text block is horizontal layout, the first segmentation axis is the horizontal candidate segmentation axis; in a case where the layout direction of the first text block is vertical layout, the first segmentation axis is the vertical candidate segmentation axis.

8. The method of any one of claims 3-7, wherein: determining the at least one candidate segmentation axis according to the fourth information comprises: obtaining an adjacency matrix of the first text block, the adjacency matrix comprising distances of adjacent words in the first text block in the vertical direction or the horizontal direction; determining a position corresponding to a maximum value in the adjacency matrix in the first text block as the candidate segmentation axis; segmenting the first text block based on a first segmentation axis of the at least one candidate segmentation axis to obtain a plurality of second text blocks comprises: clustering words on both sides of the first segmentation axis to obtain a plurality of second text blocks, wherein distances of adjacent words in the second text blocks in the vertical direction or the horizontal direction are less than the maximum value in the adjacency matrix.

9. The method of claim 8, wherein, The method further comprises: determining a position corresponding to a second maximum value in the adjacency matrix in the second text block as a second segmentation axis, the second segmentation axis penetrating the second text block; segmenting the second text block based on the second segmentation axis to obtain a plurality of third text blocks, wherein distances of adjacent words in the third text blocks in the vertical direction or the horizontal direction are less than the second maximum value in the adjacency matrix.

10. The method of claim 9, wherein, The ratio of the second maximum value to the vertical height of the smallest word in the first text block is greater than or equal to a preset threshold.

11. The method according to any one of claims 3 to 10, characterized in that, The obtaining the first text block comprises: obtaining a degree of horizontal alignment of each line of text and / or a degree of vertical alignment of each column of text in the text content; determining a plurality of continuous lines of text and / or a plurality of continuous columns of text as the first text block, wherein the degree of horizontal alignment of the lines of text is greater than a first threshold value, and the degree of vertical alignment of the columns of text is greater than a second threshold value.

12. The method according to any one of claims 2 to 11, characterized in that, The spatial information of the text includes the position of the text and the size of the text, wherein the position of the text includes the coordinates of the text in the first interface and / or the text line index of the text in the first interface.

13. The method according to any one of claims 2 to 12, characterized in that, The visual saliency feature includes at least one of the following: text color, text font, color block, shadow, frame line.

14. The method according to any one of claims 2 to 13, characterized in that, The method further comprises: In the case where the first text block is divided into a plurality of second text blocks based on a horizontal division axis, the plurality of second text blocks are sorted from top to bottom according to the spatial information of the plurality of second text blocks; or, In the case where the first text block is divided into a plurality of second text blocks based on a vertical division axis, the plurality of second text blocks are sorted from left to right according to the spatial information of the plurality of second text blocks. The sorting priority of the first text block is higher than that of the second text block.

15. The method according to any one of claims 2 to 14, characterized in that, Before the text block division of the text content according to the first information, the method further comprises: dividing all the words in the text content into a plurality of text lines, each of which extends through the first interface in the horizontal direction; sorting the plurality of text lines from top to bottom to obtain a text line index of each of the text lines, which is used for text block division of the text content and / or sorting of text blocks.

16. The method according to any one of claims 1 to 15, characterized in that, The method further comprises: According to the trajectory of the user selecting text and the first structure, determining the text block intended to be selected by the user, wherein the trajectory of the user selecting text passes through the region of at least one second text block, and the text block intended to be selected by the user includes at least one second text block; highlighting at least one second text block.

17. The method of claim 16, wherein, The method further comprises: detecting that the trajectory of the user selecting text switches from the region of the second text block to the region of the first text block, and marking the level and sorting information of the selected second text block in the first structure as a template; detecting that the trajectory of the user selecting text switches from the region of the first text block to the region of the sibling node of the first node, and determining the text block intended to be selected by the user according to the template, which includes the text block corresponding to the first node under the sibling node.

18. The method of claim 17, wherein, The method further comprises: highlighting the text block corresponding to the first node under the sibling node, wherein the visual saliency feature of the text block corresponding to the first node under the sibling node is different from that of the second text block.

19. The method of claim 17 or 18, wherein, The second operation is any one of the following: an operation of selecting text based on a cursor, an operation of selecting text based on a brush, or an operation of selecting text based on a click.

20. The method of any one of claims 16-19, wherein, The second operation is the operation of selecting text based on a click, and the determining of the text block intended to be selected by the user according to the trajectory of the user selecting the text and the first structure comprises: determining, according to the position clicked by the user and the boundary information of the text block corresponding to each node in the first structure, the lowest-level text block corresponding to the position clicked by the user; determining the lowest-level text block as the text block intended to be selected by the user.

21. The method of any one of claims 1 to 20, wherein, The method further comprises: in response to a paste operation of the user, displaying the content of at least one second text block in a second interface according to the first structure.

22. The method of claim 21, wherein, The displaying of the content of at least one second text block in a second interface according to the first structure in response to a paste operation of the user comprises: displaying the content of at least one second text block in the same row in the second interface, wherein the at least one second text block satisfies a first preset condition. The first preset condition comprises at least one of the following: the at least one second text block has the same first node; the at least one second text block has the same association relationship, and the association relationship is a subordinate relationship, a parallel relationship, or a start-end relationship.

23. The method of claim 22, wherein, The displaying of the content of at least one second text block in the second interface comprises: determining the display order of the content of at least one second text block according to seventh information, and the seventh information comprises at least one of the following information: spatial information of the at least one second text block in the first interface, semantic information of the at least one second text block, or at least one of visual saliency features related to the at least one second text block in the first interface.

24. The method of claim 21, wherein, The displaying of the content of at least one second text block in a second interface according to the first structure in response to a paste operation of the user comprises: displaying the content of at least one second text block and the content of at least one fourth text block in the second interface, and the content of the second text block and the content of the fourth text block are located in different rows in the second interface, and the second text block and the fourth text block satisfy a second preset condition. The second preset condition comprises at least one of the following: the second text block and the fourth text block have different first nodes; the second text block and the fourth text block have different association relationships, and the association relationship comprises a subordinate relationship, a parallel relationship, or a start-end relationship.

25. The method of claim 24, wherein, The displaying of the content of at least one second text block and the content of at least one fourth text block in the second interface comprises: determining the display order of the content of at least one second text block and the content of at least one fourth text block according to eighth information, and the eighth information comprises at least one of the following information: spatial information of the at least one second text block in the first interface, hierarchical information of the at least one second text block, spatial information of the at least one fourth text block in the first interface, or hierarchical information of the at least one fourth text block.

26. The method of any one of claims 21-25, wherein, The method further includes: displaying a format mark in the second interface, the format mark being used to identify the association between at least one of the second text blocks and / or the hierarchical relationship of at least one of the second text blocks in the first structure.

27. The method of claim 26, wherein, The format mark includes at least one of the following: parentheses, semicolon, dash, serial number, bullet, line break, tab, and vertical bar.

28. The method of any one of claims 1 to 27, wherein, The method further includes: detecting an operation of the user to format the content of the second text block, and adjusting the display format of the text blocks in the same hierarchy as the second text block in the second interface in batches.

29. An electronic device, comprising: Comprise: one or more processors; one or more memories; The one or more memories store one or more computer programs, the one or more computer programs comprising instructions that, when executed by the one or more processors, cause the electronic device to perform the method of any one of claims 1-28.

30. An apparatus for human-computer interaction, the apparatus comprising: Comprise modules for implementing the method of any one of claims 1-28.

31. A computer readable storage medium, characterized in that, The storage medium stores programs or instructions, which, when run on a computer, cause the method of any one of claims 1-28 to be performed.

32. A chip, comprising: The chip stores instructions, which, when run, cause the method of any one of claims 1-28 to be performed.

33. A computer program product, characterised in that, The computer program product stores programs or instructions, which, when run, cause the method of any one of claims 1-28 to be performed.

Images