Human-computer interaction method and device and electronic equipment

Abstract

The invention provides a man-machine interaction method and device and electronic equipment. In the man-machine interaction method, in response to a text selection operation of a user, the electronic equipment can identify text content displayed in a first interface, and generate a tree structure of the text content according to an identification result. Each node in the tree structure is a text block, and the text block of the lower child node is located in the text block range of the upper father node. The user selects the text by taking the text block as a unit, and the electronic equipment highlights the text block selected by the user by taking the text block as the unit according to the tree structure. And when the user pastes the text, the electronic equipment displays the content of the selected text block on the second interface according to the hierarchy of the tree structure. According to the technical scheme, based on the constructed tree structure, the text selection efficiency, the user selection intention and the reading efficiency after pasting can be considered, and the interaction experience in the text selection process can be improved.

Description

[0001] This application is a divisional application. The application number of the original application is 202411331279.1, and the application date of the original application is September 23, 2024. The entire content of the original application is incorporated herein by reference. Technical Field

[0002] This application relates to the technical field of electronic devices, and more particularly, to a human-computer interaction method, apparatus, and electronic device. Background Art

[0003] With the continuous development of electronic device technology, the functions of electronic devices are becoming increasingly rich. To meet users' needs for sharing and reusing text information, many electronic devices have a text extraction function to facilitate users to select and paste text information that cannot be directly edited in the interface. For example, users can select text information by moving the cursor, smearing, or clicking. However, based on the current solutions, when users use the text extraction function, either the operation efficiency is low, the selection intention cannot be met, or it is difficult to read after pasting, resulting in a poor interaction experience. Summary of the Invention

[0004] This application provides a human-computer interaction method, apparatus, and electronic device, which can improve the interaction experience during the text extraction process.

[0005] In a first aspect, a human-computer interaction method is provided, including: displaying a first interface, where the first interface includes text content; in response to a first operation of a user, identifying the text content; according to the result of identifying the text content, generating a tree structure of the text content, where the tree structure is used to represent the hierarchical information of the text content, and a parent node in the tree structure corresponds to a first text block, and a child node of the parent node corresponds to a second text block, and the second text block is located within the area of the first text block; in response to a second operation of the user, highlighting at least one second text block according to the tree structure.

[0006] In the embodiments of this application, by constructing a tree structure of the text content in the first interface, the text content in the first interface can be hierarchically organized in the form of text blocks, thereby providing the correct reading order and supporting multiple reading orders. Correspondingly, when extracting text, the order in which users select and paste text is more in line with the general reading habit, so that the selection intention of users can be met, the reading efficiency and the integrity of information dissemination can be improved. Moreover, the method of users selecting text based on text blocks is simple in operation and high in selection efficiency. Therefore, the solution provided by this application can take into account the selection efficiency, the selection intention of users, and the reading efficiency, thus significantly improving the interaction experience of users.

[0007] In combination with the first aspect, in a possible implementation manner, a tree structure of the text content is generated according to the result of recognizing the text content, including: dividing the text content into text blocks according to first information, where the first information includes at least one of spatial information of the text, semantic information of the text, and visual saliency features in the first interface; determining a hierarchical relationship between the divided text blocks according to second information, where the second information includes at least one of spatial information of the text, semantic information of the text, and visual saliency features in the first interface; sorting the divided text blocks according to third information, where the third information includes spatial information of the text.

[0008] Generating a tree structure based on the first information, the second information, and the third information can achieve precise segmentation of the original layout text and sorting that conforms to the general reading order. In the process of constructing the tree structure, compared with layout analysis that completely relies on image information, layout analysis based on text position information has stronger real-time performance.

[0009] In combination with the first aspect, in a possible implementation manner, dividing the text content into text blocks according to the first information includes: obtaining the first text block; determining at least one candidate segmentation axis according to fourth information, where the candidate segmentation axis penetrates the first text block; determining the layout direction of the first text block according to the at least one candidate segmentation axis, where the layout direction of the first text block is horizontal layout or vertical layout; dividing the first text block based on a first segmentation axis among the at least one candidate segmentation axes to obtain a plurality of second text blocks, where the extension direction of the first segmentation axis is the same as the layout direction of the first text block.

[0010] Specifically, when the layout direction of the first text block is horizontal layout, the first segmentation axis extends horizontally. When the layout direction of the first text block is vertical layout, the first segmentation axis extends vertically.

[0011] Using the layout direction of the first text block to screen out the segmentation axis actually used for segmentation from the at least one candidate segmentation axis can eliminate inappropriate segmentation axes and avoid unreasonable segmentation methods.

[0012] In combination with the first aspect, in a possible implementation manner, the fourth information includes at least one of the following information: visual saliency features in the first interface, the alignment degree of the text horizontally, the alignment degree of the text vertically, the mapping relationship between the semantic information of the text and the spatial information of the text, and the arrangement density of the text.

[0013] There are many ways to obtain candidate segmentation axes, which can achieve precise segmentation of the original layout text.

[0014] In combination with the first aspect, in a possible implementation, the horizontal alignment degree of the text is determined according to at least one of the differences in inter-character size, the difference in the upper boundary of the inter-character, the difference in the lower boundary of the inter-character, and the difference in the ordinate of the center point of the inter-character in the text; and / or the vertical alignment degree of the text is determined according to at least one of the differences in inter-character size, the difference in the left boundary of the inter-character, the difference in the right boundary of the inter-character, and the difference in the abscissa of the center point of the inter-character in the text.

[0015] In combination with the first aspect, in a possible implementation, at least one candidate segmentation axis includes one or more horizontal candidate segmentation axes and one or more vertical candidate segmentation axes. Determining the layout direction of the first text block according to at least one candidate segmentation axis includes: pre-segmenting the first text block based on one or more horizontal candidate segmentation axes to obtain a plurality of line text areas; determining the similarity of the line layout of the first text block according to the fifth information, where the fifth information includes at least one of the difference in the alignment method between line text areas, the difference in line width between line text areas, the difference in the number of texts between line text areas, the difference in the text size of the same column between line text areas, and the difference in line spacing between line text areas; pre-segmenting the first text block based on one or more vertical candidate segmentation axes to obtain a plurality of column text areas; determining the similarity of the column layout of the first text block according to the sixth information, where the sixth information includes at least one of the difference in the alignment method between column text areas, the difference in column width between column text areas, the difference in the number of texts between column text areas, the difference in the text size of the same row between column text areas, and the difference in column spacing between column text areas; determining the layout direction of the first text block according to the similarity of the line layout and the similarity of the column layout, where, when the similarity of the line layout is greater than the similarity of the column layout, the layout direction of the first text block is horizontal layout; when the similarity of the column layout is greater than the similarity of the line layout, the layout direction of the first text block is vertical layout.

[0016] Determining the layout direction of the text block by calculating the layout similarity can segment the text block in a way that conforms to the general reading order.

[0017] In combination with the first aspect, in a possible implementation, before segmenting the first text block based on the first segmentation axis in at least one candidate segmentation axis, the method further includes: determining the first segmentation axis from at least one candidate segmentation axis according to the layout direction of the first text block, where, when the layout direction of the first text block is horizontal layout, the first segmentation axis is the horizontal candidate segmentation axis; when the layout direction of the first text block is vertical layout, the first segmentation axis is the vertical candidate segmentation axis.

[0018] Filtering out the splitting axes actually used for splitting from at least one candidate splitting axis according to the layout direction of the first text block can eliminate inappropriate splitting axes, so as to split the text block in a manner consistent with the general reading order.

[0019] In combination with the first aspect, in a possible implementation manner, determining at least one candidate splitting axis according to the fourth information includes: obtaining the adjacency matrix of the first text block, where the adjacency matrix includes the distances in the vertical or horizontal directions between adjacent words in the first text block; determining the position corresponding to the maximum value in the adjacency matrix in the first text block as the candidate splitting axis; splitting the first text block based on the first splitting axis in at least one candidate splitting axis to obtain a plurality of second text blocks, including: clustering the words on both sides of the first splitting axis to obtain a plurality of second text blocks, where the distances in the vertical or horizontal directions between adjacent words in the second text blocks are less than the maximum value in the adjacency matrix.

[0020] Dividing the text block based on the adjacency matrix can discover clusters of any shape and is insensitive to outliers in the dataset.

[0021] In combination with the first aspect, in a possible implementation manner, the method further includes: determining the position corresponding to the second largest value in the adjacency matrix in the second text block as the second splitting axis, and the second splitting axis penetrates the second text block; splitting the second text block based on the second splitting axis to obtain a plurality of third text blocks, where the distances in the vertical or horizontal directions between adjacent words in the third text blocks are less than the second largest value in the adjacency matrix.

[0022] Based on the adjacency matrix, all clusters can be discovered in one algorithm execution without the need to pre-specify the number of clusters.

[0023] In combination with the first aspect, in a possible implementation manner, the ratio of the second largest value to the vertical height of the smallest word in the first text block is greater than or equal to a preset threshold.

[0024] When the distance between adjacent words is less than the preset threshold compared to the size of the smallest word in the first text block, it can be considered that the adjacent words have an associated relationship and should not be split into different text blocks.

[0025] In combination with the first aspect, in a possible implementation manner, obtaining the first text block includes: obtaining the horizontal alignment degree of each line of text in the text content and / or the vertical alignment degree of each column of text; determining a plurality of consecutive text lines and / or a plurality of consecutive text columns as the first text block, where the horizontal alignment degree of the text line is greater than a first threshold, and the vertical alignment degree of the text column is greater than a second threshold.

[0026] In combination with the first aspect, in a possible implementation, the spatial information of the text includes the position and size of the text, where the position of the text includes the coordinates of the text in the first interface and / or the text line index of the text in the first interface.

[0027] Performing layout analysis based on the spatial position of the text has stronger real-time performance.

[0028] In combination with the first aspect, in a possible implementation, the visual saliency feature includes at least one of the following: text color, text font, color block, shadow, box line.

[0029] In combination with the first aspect, in a possible implementation, according to the third information, sorting the divided text blocks includes: in the case where the first text block is divided based on a horizontal splitting axis to obtain multiple second text blocks, sorting the multiple second text blocks from top to bottom according to the spatial information of the multiple second text blocks; or, in the case where the first text block is divided based on a vertical splitting axis to obtain multiple second text blocks, sorting the multiple second text blocks from left to right according to the spatial information of the multiple second text blocks; where the sorting priority of the first text block is higher than that of the second text block.

[0030] For the text blocks obtained by horizontal splitting, the direction of sorting these text blocks from top to bottom is consistent with the typesetting direction of the divided text block, which conforms to the reading order of horizontal typesetting. For the text blocks obtained by vertical splitting, the direction of sorting these text blocks from left to right is consistent with the typesetting direction of the divided text block, which conforms to the reading order of vertical typesetting.

[0031] In combination with the first aspect, in a possible implementation, before dividing the text content into text blocks according to the first information, the method further includes: dividing all the words and phrases in the text content into multiple text lines, each text line running horizontally through the first interface; sorting the multiple text lines from top to bottom to obtain the text line index of each text line, and the text line index is used for dividing the text content into text blocks and / or sorting the text blocks.

[0032] The text line index is the global line number of the text in the first interface and can be used to judge the relative position between text blocks.

[0033] In combination with the first aspect, in a possible implementation, in response to the user's second operation, highlighting at least one second text block according to the tree structure includes: determining the text block that the user intends to select according to the user's text selection trajectory and the tree structure, where the user's text selection trajectory passes through the area of at least one second text block, and the text block that the user intends to select includes at least one second text block; highlighting at least one second text block.

[0034] Based on the tree structure, the text block intended to be selected by the user can be determined according to the trajectory of the user's text selection. When the user selects text, the text structure can be used as a unit for efficient text selection that conforms to the general reading flow, without the problem of interspersed selection or disordered selection between different structures. When there are multiple possible arrangements of text blocks, the trajectory of the user's text selection can freely express the text selection order that conforms to their intention.

[0035] In combination with the first aspect, in a possible implementation manner, determining the text block intended to be selected by the user according to the trajectory of the user's text selection and the tree structure includes: detecting that the trajectory of the user's text selection switches from the area of the second text block to the area of the first text block, and marking the level and sorting information of the second text block selected in the first text block in the tree structure as a template; detecting that the trajectory of the user's text selection switches from the area of the first text block to the area of the sibling node of the parent node, and determining that the text block intended to be selected by the user includes the text blocks corresponding to the child nodes under the sibling node according to the template.

[0036] In this way, speculating the text block intended to be selected according to the user's selection trajectory can better meet the user's selection intention and improve the selection efficiency.

[0037] In combination with the first aspect, in a possible implementation manner, the method further includes: highlighting the text blocks corresponding to the child nodes under the sibling node, where the visual saliency feature of the text blocks corresponding to the child nodes under the sibling node is different from the visual saliency feature of the second text block.

[0038] The visual saliency feature of the text block selected by mapping in the second interface is different from the visual saliency feature of the second text block in the second interface, which can prompt the user.

[0039] In combination with the first aspect, in a possible implementation manner, the second operation is any one of the following: an operation of selecting text based on the cursor, an operation of selecting text based on smearing, or an operation of selecting text based on clicking.

[0040] In combination with the first aspect, in a possible implementation manner, the second operation is an operation of selecting text based on clicking. Determining the text block intended to be selected by the user according to the trajectory of the user's text selection and the tree structure includes: determining the text block at the lowest level corresponding to the position where the user clicks according to the position where the user clicks and the boundary information of the text blocks corresponding to each node in the tree structure; determining the text block at the lowest level as the text block intended to be selected by the user.

[0041] In this way, the user can select a line of ordinary text or the information of an item with one click, improving the interaction efficiency.

[0042] In combination with the first aspect, in a possible implementation, the method further includes: in response to a paste operation by the user, displaying the content of at least one second text block on the second interface according to the tree structure.

[0043] When pasting, the electronic device displays at least one second text block on the second interface according to the levels of the tree structure.

[0044] In the embodiments of the present application, the tree structure constructed for the text content can provide the user with a general reading order. Therefore, when the user pastes and displays the content of the selected text block on the second interface, the selected text block will also display relevant information in an order that conforms to the reading habit.

[0045] In combination with the first aspect, in a possible implementation, in response to a paste operation by the user, displaying the content of at least one second text block on the second interface according to the tree structure includes: displaying the content of at least one second text block in the same row on the second interface, where at least one second text block meets a first preset condition; the first preset condition includes at least one of the following: at least one second text block has the same parent node; there is the same association relationship between at least one second text block, and the association relationship is a subordinate relationship, a parallel relationship, or a start-end relationship.

[0046] The child nodes with the same parent node usually have a relatively high degree of proximity in position and will appear in the user's small-scale reading hot zone at the same time. Displaying the text blocks with the same parent node in one row conforms to the user's reading order. The text blocks belonging to the same association relationship are close in reading order. Displaying the text blocks belonging to the same association relationship together conforms to the reading order and can improve the reading efficiency.

[0047] In combination with the first aspect, in a possible implementation, displaying at least one second text block in the same row on the second interface includes: determining the display order of the content of at least one second text block according to seventh information, where the seventh information includes at least one of the following information: the spatial information of at least one second text block in the first interface, the semantic information of at least one second text block, or at least one of the visual saliency features related to at least one second text block in the first interface.

[0048] When different text blocks are displayed in the same row, the order of the text blocks can be rearranged, which can improve the reading efficiency.

[0049] In combination with the first aspect, in a possible implementation manner, in response to a user's paste operation, the content of at least one second text block is displayed on a second interface according to the tree structure, including: displaying the content of at least one second text block and the content of at least one fourth text block on the second interface, where the content of the second text block and the content of the fourth text block are on different lines of the second interface, and the second text block and the fourth text block meet a second preset condition; the second preset condition includes at least one of the following: the second text block and the fourth text block have different parent nodes; the second text block and the fourth text block have different association relationships, and the association relationships include a subordinate relationship, a parallel relationship, and a start-end relationship.

[0050] Displaying child nodes with different parent nodes or child nodes with different association relationships on separate lines can highlight the structural hierarchy between texts and help improve the information dissemination efficiency.

[0051] In combination with the first aspect, in a possible implementation manner, displaying the content of at least one second text block and the content of at least one fourth text block on the second interface includes: determining the display order of the content of at least one second text block and the content of at least one fourth text block according to eighth information, where the eighth information includes at least one of the following information: the spatial information of at least one second text block in the first interface, the hierarchical information of at least one second text block, the spatial information of at least one fourth text block in the first interface, or the hierarchical information of at least one fourth text block.

[0052] In this application, reordering according to the position information and hierarchical information of text blocks helps to restore the structural hierarchy and association relationships contained in the original layout and improve the reading efficiency.

[0053] In combination with the first aspect, in a possible implementation manner, the method further includes: displaying format markers on the second interface, where the format markers are used to identify the association relationships between at least one second text block and / or the hierarchical relationships of at least one second text block in the tree structure.

[0054] Adding special symbols helps to restore the structural hierarchy and association relationships contained in the original layout and improves the efficiency and integrity of information dissemination.

[0055] In combination with the first aspect, in a possible implementation manner, the format markers include at least one of the following: parentheses, semicolons, dashes, serial numbers, bullet points, line breaks, tab stops, vertical bar symbols.

[0056] In combination with the first aspect, in a possible implementation manner, the method further includes: detecting an operation by the user to adjust the format of the content of the second text block, and batch-adjusting the display formats of the text blocks at the same level as the second text block on the second interface.

[0057] The electronic device can provide the user with the function of batch-adjusting the display formats of text blocks at the same level, which can improve the efficiency of format adjustment.

[0058] In combination with the first aspect, in a possible implementation, the granularity of the second text block is smaller than that of the first text block.

[0059] In combination with the first aspect, in a possible implementation, among the multiple child nodes of the parent node, adjacent child nodes are interconnected bidirectionally.

[0060] In combination with the first aspect, in a possible implementation, the leaf nodes in the tree structure correspond to words or text lines.

[0061] In combination with the first aspect, in a possible implementation, the first interface displays at least one of the following: pictures, web pages, or documents.

[0062] In combination with the first aspect, in a possible implementation, the text content is not editable.

[0063] In a second aspect, there is provided an apparatus, which is included in an electronic device and has the function of implementing the actions involved in the above-mentioned first aspect and any possible implementation of the first aspect.

[0064] This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a display module or unit, an identification module or unit, a generation module or unit, a division module or unit, a sorting module or unit, an acquisition module or unit, a detection module or unit, a processing module or unit, etc.

[0065] In a third aspect, there is provided an electronic device, including: one or more processors; one or more memories; the one or more memories store one or more computer programs, and the one or more computer programs include instructions that, when executed by the one or more processors, cause the electronic device to execute the methods in the above-mentioned first aspect and any possible implementation of the first aspect.

[0066] In a fourth aspect, there is provided a computer-readable storage medium, including computer instructions that, when running on an electronic device, cause the electronic device to execute the methods in the above-mentioned first aspect and any possible implementation of the first aspect.

[0067] In a fifth aspect, there is provided a computer program product containing instructions that, when the computer program product runs on a computer, cause the computer to execute the methods in the above-mentioned first aspect and any possible implementation of the first aspect.

[0068] In a sixth aspect, a chip is provided. The chip includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface and executes the methods in the first aspect and any possible implementation manner of the first aspect.

[0069] Optionally, as an implementation manner, the chip may further include a memory. Instructions are stored in the memory, and the processor is configured to execute the instructions stored in the memory. When the instructions are executed, the processor is configured to execute the methods in the first aspect and any possible implementation manner of the first aspect.

[0070] The above-mentioned chip may specifically be a field-programmable gate array or an application-specific integrated circuit.

[0071] The beneficial effects of the devices described in the second aspect to the sixth aspect may refer to the beneficial effects of the methods described in the first aspect, and will not be elaborated herein. Description of the Drawings

[0072] Figure 1 is a schematic hardware structure diagram of an electronic device provided by an embodiment of the present application.

[0073] Figure 2 is a schematic software structure diagram of an electronic device provided by an embodiment of the present application.

[0074] Figures 3 - 5 is a schematic interface diagram of the existing text extraction effect.

[0075] Figure 6 is a schematic flowchart of a human-computer interaction method provided by an embodiment of the present application.

[0076] Figure 7 is a partial schematic diagram of a tree structure provided by an embodiment of the present application.

[0077] Figure 8 is a partial schematic diagram of a tree structure provided by an embodiment of the present application.

[0078] Figure 9 is a schematic flowchart of generating a tree structure in the human-computer interaction method provided by an embodiment of the present application.

[0079] Figures 10 - 13 is a schematic diagram of determining the position of the splitting axis in the human-computer interaction method provided by the present application.

[0080] Figure 14 is a schematic diagram of word features provided by an embodiment of the present application.

[0081] Figure 15 is a schematic diagram of the layout direction of a text block provided by an embodiment of the present application.

[0082] Figures 16 - 18 It is a schematic diagram for determining the typesetting direction in the human-computer interaction method provided by an embodiment of this application.

[0083] Figures 19 - 21 It is a schematic diagram for selecting text in the human-computer interaction method provided by an embodiment of this application.

[0084] Figure 22 It is a partial schematic diagram of a tree structure provided by an embodiment of this application.

[0085] Figures 23 - 24 It is a schematic diagram for selecting text in the human-computer interaction method provided by an embodiment of this application.

[0086] Figures 25 - 29 It is a schematic diagram for pasting text in the human-computer interaction method provided by an embodiment of this application.

[0087] Figure 30 It is a schematic structural block diagram of a device provided by an embodiment of this application.

[0088] Figure 31 It is a schematic structural block diagram of an electronic device provided by an embodiment of this application. Detailed implementation manners

[0089] Next, the technical solutions in this application will be described with reference to the accompanying drawings.

[0090] It should be noted that in the description of the embodiments of this application, unless otherwise specified, " / " means "or". For example, A / B may mean A or B; herein, "and / or" is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0091] In the embodiments of this application, terms such as "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first", "second", etc. may explicitly or implicitly include one or more of such features. In addition, in the description of the embodiments of this application, "a plurality" means two or more than two, "at least one" and "one or more" mean one, two or more than two. The singular forms "a", "one kind", "the", "above-mentioned", "this" and "this one" are also intended to include expressions such as "one or more", unless there is a clear opposite indication in the context.

[0092] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment," "in some embodiments," "in other some embodiments," "in still other embodiments," etc., which appear in different places in this specification, do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and their variants mean "including but not limited to," unless otherwise specifically emphasized.

[0093] For ease of understanding, some technical terms involved in the embodiments of the present application will be explained and described below.

[0094] Tree structure: A hierarchical structure defined by a branching relationship, which is a finite set of n nodes. The basic unit of a tree structure is a node, the link between nodes is called a branch, and the nodes and branches form a tree shape.

[0095] Root node: The topmost node, which is the beginning of the tree structure.

[0096] Leaf node (or terminal node): The outermost node, which is the end of the tree structure. A leaf node has only a predecessor node and no successor node.

[0097] Parent node: The node on the upper layer of a certain node (i.e., the predecessor node) is called the parent node of that node.

[0098] Child node: The node on the lower layer of a certain node (i.e., the successor node) is called the child node of that node.

[0099] Sibling nodes: Nodes that have the same parent node are sibling nodes to each other.

[0100] Ancestor node: All nodes on the branches from the root node to a certain node are called the ancestor nodes of that node.

[0101] Descendant node: Any node in the subtree rooted at a certain node is called a descendant node of that node.

[0102] Cousin nodes: Nodes whose parent nodes are on the same layer are cousin nodes to each other.

[0103] Subtree: A tree formed with the child nodes of a certain node as the root is called the subtree of that node.

[0104] Level of a node: Starting from the root node, the root node is the first layer, the children of the root are the second layer. If a certain node is on the L-th layer, then the root of its subtree is on the (L + 1)-th layer.

[0105] Parent-child relationship: In a hierarchy, the relationship between a node and its direct child nodes is defined as the parent-child relationship. There is an inclusion relationship between a parent node and its child nodes, where the parent node contains the information of its child nodes.

[0106] Sibling relationship: Sibling nodes refer to all child nodes that have the same parent node. There is an equivalence relationship between these nodes, meaning they share the same parent node but do not have an inclusion relationship with each other.

[0107] Superior-subordinate relationship: In a hierarchy, the superior-subordinate relationship refers to the relationship between a node and its indirect parent node. This relationship is usually used to represent the connection between entities with multi-level inclusion relationships.

[0108] Visual saliency feature: It refers to the area or object in a visual scene that can attract people's attention. For real-world scenes, people will automatically identify the regions of interest and process the regions of interest while ignoring the regions they are not interested in. The definition of visual saliency in computer vision is to use computer technology to simulate the perception process of the human visual system for images, extract the salient regions in the images (i.e., the regions of interest to humans), so as to quickly search for and locate the target of interest when facing natural scenes.

[0109] Density-based spatial clustering of applications with noise (DBSCAN): Defines a cluster as the largest set of density-connected points, capable of dividing regions with sufficient high density into clusters and discovering clusters of arbitrary shapes in a spatial database with noise.

[0110] Optical character recognition (OCR): It refers to the technology of scanning and analyzing the text in an image or document and converting it into editable and searchable data.

[0111] Layout: It refers to the design, typesetting, and organization method of a page, including the position and distribution of elements such as text and pictures.

[0112] The method provided by the embodiments of the present application can be applied to an electronic device with a display function. For example, it can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and smart home devices (such as smart screens). The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0113] Exemplarily, Figure 1 FIG. shows a schematic hardware structure diagram of an electronic device provided by the embodiments of the present application.

[0114] As Figure 1 shown, the electronic device 100 may include: a processor 110, a memory 120, a universal serial bus (USB) interface 130, a power supply 140, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a microphone 170C, a sensor module 180, a camera 191, a display screen 192, etc.

[0115] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0116] Among them, the controller may be the nerve center and command center of the electronic device. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching instructions and executing instructions.

[0117] A memory can also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from this memory, avoiding repeated accesses, reducing the waiting time of the processor 110, and thus improving the efficiency of the system.

[0118] The NPU is a neural-network (NN) computing processor. By learning from the structure of biological neural networks, such as the transmission mode between human brain neurons, it can quickly process input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0119] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0120] For example, the processor 110 and the touch sensor 180B can communicate through the I2C bus interface to implement the touch function of the electronic device. The processor 110 and the camera 191 can communicate through the CSI interface to implement the shooting function of the electronic device. The processor 110 and the display screen 192 can communicate through the DSI interface to implement the display function of the electronic device.

[0121] It can be understood that the interface connection relationships between the modules illustrated in the embodiments of the present application are only illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0122] The power supply 140 is used to supply power to the processor 110, the memory 120, the display screen 192, the camera 191, the mobile communication module 150, the wireless communication module 160, etc.

[0123] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, the baseband processor, etc.

[0124] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0125] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc. applied to the electronic device. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through the antenna 1 and radiate it out. In some embodiments, at least some functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 can be disposed in the same device.

[0126] The wireless communication module 160 may provide solutions for wireless communications applied to an electronic device, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 for radiation.

[0127] The electronic device 100 realizes the display function through the GPU, the display screen 192, and the application processor, etc.

[0128] The GPU is a microprocessor for image processing, connected to the display screen 192 and the application processor. The GPU is used to execute mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0129] The display screen 192 is used to display images, videos, etc. The display screen 192 includes a display panel. The display panel may adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 192, where N is a positive integer greater than 1.

[0130] The electronic device 100 can implement the shooting function through the ISP, camera 191, video codec, GPU, display screen 192, application processor, etc.

[0131] The ISP is used to process the data fed back by the camera 191. The camera 191 is used to capture still images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. In some embodiments, the electronic device may include one or N cameras 191, where N is a positive integer greater than 1.

[0132] The video codec is used to compress or decompress digital videos. The electronic device can support one or more video codecs. In this way, the electronic device can play or record videos in multiple encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0133] The memory 120 is used to store data and / or instructions.

[0134] The memory 120 may include an internal memory. The internal memory is used to store computer-executable program code, and the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory. The internal memory may include a program storage area and a data storage area. Among them, the program storage area can store the operating system; the program storage area can also store one or more application programs (such as a gallery, contacts, etc.). The data storage area can store the data created during the use of the electronic device (such as images, contacts, etc.). In addition, the internal memory may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, universal flash storage (UFS), etc. In some embodiments, the processor 110 can make the electronic device execute the methods provided in the embodiments of the present application by running the instructions stored in the internal memory and / or the instructions stored in the memory provided in the processor 110.

[0135] The memory 120 may also include an external memory, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory can communicate with the processor 110 through an external memory interface to implement the data storage function. For example, save files such as music and videos in the external memory.

[0136] The electronic device can implement audio functions such as audio playback and recording through the audio module 170, speaker 170A, microphone 170C, and application processor, etc.

[0137] The sensor module 180 may include a pressure sensor 180A, a touch sensor 180B, a gyro sensor, an acceleration sensor, a distance sensor, and some other sensors, etc.

[0138] The pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be disposed on the display screen 192. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor may include at least two parallel plates with conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 192, the electronic device detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations with the same touch position but different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0139] The touch sensor 180B is also called a "touch panel". The touch sensor 180B may be disposed on the display screen 192, and the touch sensor 180B and the display screen 192 form a touch screen, also called a "touch control screen". The touch sensor 180B is used to detect touch operations acting on or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 192. In some other embodiments, the touch sensor 180B may also be disposed on the surface of the electronic device, at a different position from the display screen 192.

[0140] The gyro sensor can be used to determine the motion posture of the electronic device. For example, the gyro sensor can be used for anti-shake shooting and can also be used in navigation, somatosensory game scenarios, etc. When the electronic device is stationary, the gyro sensor can detect the magnitude and direction of gravity.

[0141] The acceleration sensor can detect the magnitude of the acceleration of the electronic device in various directions (generally three axes).

[0142] A distance sensor for measuring distance. The electronic device can measure distance through infrared or laser. In some embodiments, when shooting a scene, the electronic device can use the distance sensor to measure distance to achieve fast focusing.

[0143] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0144] The above introduced the schematic diagram of the possible hardware structure of the electronic device. The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. The embodiments of the present application take the system as an example to exemplarily illustrate the software structure of the electronic device. However, it can be understood that the software system of the electronic device in the embodiments of the present application can also be other systems, such as HarmonyOS, system, system, etc., which will not be elaborated one by one here.

[0145] Figure 2 Shows the software structure block diagram of an electronic device provided by the embodiments of the present application. As Figure 2 shown, the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, from top to bottom are the application layer, the application framework layer, the system runtime library layer, and the kernel layer.

[0146] The application layer may include a series of application packages (application). As Figure 2 shown, the application packages may include applications such as camera, gallery, calendar, call, map, weather, WLAN, Bluetooth, music, video, short message, voice assistant, etc. The applications are mainly related to the user interface (UI), and are usually written in JAVA language by calling the interfaces of the application framework layer.

[0147] The application framework layer provides application programming interfaces (API) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions. As Figure 2 shown, the application framework layer may include a window manager, a content provider, a telephone manager, a resource manager, a notification manager, a view system, etc.

[0148] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0149] The content provider is used to store and obtain data, and make this data accessible to application programs. The data may include videos, images, audio, dialed and received calls, browsing history and bookmarks, phone books, etc.

[0150] The phone manager is used to provide the communication function of the electronic device. For example, the management of call status (including connection, hangup, etc.).

[0151] The resource manager provides various resources for application programs, such as localized strings, icons, pictures, layout files, video files, etc.

[0152] The notification manager enables application programs to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform the completion of a download, message reminder, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as the notification of a background-running application program, and can also be a notification that appears on the screen in the form of a dialogue window. For example, prompt text information in the status bar, emit a prompt sound, the terminal device vibrates, the indicator light flashes, etc.

[0153] The view system includes visible controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build application programs. The display interface can be composed of one or more views. For example, the display interface including the SMS notification icon can include a view for displaying text and a view for displaying pictures.

[0154] The system runtime library layer is a collection of a series of program libraries located below the application framework layer, which can be divided into two parts, namely system libraries and application runtime (such as Android runtime).

[0155] The application runtime includes core libraries and a virtual machine. The application runtime is responsible for the scheduling and management of the software system. The core libraries contain two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core libraries of the software system.

[0156] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as the management of object life cycles, stack management, thread management, security and exception management, and garbage collection.

[0157] The system library is the support of the application framework and can include multiple functional modules, such as: the surface manager, media libraries, 2D graphics engine (such as SGL), 3D graphics processing library (such as OpenGL ES), image processing library, etc.

[0158] The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0159] The media libraries support the playback and recording of multiple common audio and video formats, as well as static image files, etc. The media libraries can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0160] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0161] The 2D graphics engine is a drawing engine for 2D drawing.

[0162] The kernel layer is the layer between the hardware and software, and is used to provide the essential functions of the operating system, such as file management, memory management, process management, network protocol stack, etc. The kernel layer at least includes a display driver, a camera driver, an audio driver, a sensor driver, a Bluetooth driver, etc.

[0163] For ease of understanding, the following embodiments of the present application will take an electronic device with the Figure 1 and Figure 2 shown structure as an example, and in combination with the accompanying drawings and application scenarios, specifically elaborate on the human-computer interaction method provided by the embodiments of the present application.

[0164] With the continuous development of electronic device technology, the functions of electronic devices are becoming more and more abundant. In order to meet the user's needs for sharing and reusing text information, many electronic devices have a text extraction function to facilitate the user to select and paste text information that cannot be directly edited in the interface. For example, the user can select text information by moving the cursor, smearing, or clicking, etc. However, based on the current solution, the user experience is poor when using the text extraction function.

[0165] For example, in the method of text selection based on the cursor, situations such as multiple selection, missed selection, and wrong selection often occur. Figure 3 Shows a schematic diagram of the interface for text selection based on the cursor in the existing solution.

[0166] Refer to Figure 3As shown in (a) therein, the content within the dashed box is the text that the user intends to select (such as multiple trip information). However, when dragging the cursor to select the text, only part of the text is selected, resulting in the lack of partial information and causing under-selection.

[0167] Refer to Figure 3 As shown in (b) therein, the content within the dashed box is the text that the user intends to select. However, when dragging the cursor, the cursor jumps randomly. For example, the position of the cursor being adjusted (i.e., dragged) jumps, or the position of the cursor not being adjusted (i.e., not dragged) jumps, such as the situation where the start cursor and the end cursor shown in the figure are reversed (i.e., the front and back order is reversed), making it difficult for the user to control and thus resulting in the selection of incorrect content.

[0168] Refer to Figure 3 As shown in (c) and (d) therein, the content within the dashed box is the text that the user intends to select. However, when dragging the cursor to select the text within the dashed box, other text outside the dashed box is also selected, resulting in over-selection.

[0169] The method of selecting text based on the cursor depends on the system's own sorting of the text. The occurrence of the above-mentioned under-selection and missed selection is mainly because the system's layout analysis of the page is incorrect, resulting in the disorder of text sorting, and the user cannot select text according to the general reading order. The occurrence of the above-mentioned over-selection is, on the one hand, because the system's layout analysis of the page is incorrect, resulting in the disorder of text sorting, as Figure 3 shown in (c) therein; on the other hand, because the system's layout analysis result of the page only supports one reading order, so the user can only select text in one order. Taking Figure 3 shown in (d) therein as an example, there are actually two potential reading orders for the itinerary section on the page. One is the horizontal reading order, paying attention to both the outbound trip and the return trip at the same time; the other is the vertical reading order, only paying attention to the outbound trip or the return trip. If the system only supports the horizontal reading order, when the user intends to select the content in the dashed box 1, the return trip information will be over-selected; if the system only supports the vertical reading order, when the user intends to select the content in the dashed box 2, the outbound trip information will be over-selected.

[0170] All in all, the method of selecting text based on the cursor cannot well meet the user's selection intention, and the interaction experience is poor.

[0171] For another example, in the method of text selection based on smearing or clicking, the user needs to select or deselect each word one by one, and the operation efficiency is low. Figure 4 shows a schematic diagram of the interface for text selection based on smearing or clicking in the existing solution.

[0172] Refer to Figure 4As shown in (a) of [reference], in one solution, the system separately displays the text content on the page on a card, and the user clicks on words one by one or selects words by smearing in any order. In this way, the card covers a part of the page area, and the text layout in the card is inconsistent with the layout of the original page. Therefore, when selecting and adjusting the text, the user needs to screen the text against the original image and needs to select or cancel item by item, which significantly affects the operation efficiency and increases the cognitive load of the interaction.

[0173] Reference Figure 4 As shown in (b) of [reference], in another solution, the system highlights a text box on the original page, and the user clicks on words one by one or selects words by smearing in any order. In this way, the user needs to accurately select or cancel, and the small-scale clicking or smearing exacerbates the complexity of the operation, requiring the user to operate precisely, resulting in low efficiency.

[0174] All in all, the method of selecting text by smearing or clicking is complex in operation, low in selection efficiency, and poor in interaction experience.

[0175] In addition, after selecting the text, there is also a problem of poor interaction experience in the process of pasting the text to other positions. Figure 5 Shows a schematic diagram of the interface for text selection and pasting in the existing solution.

[0176] Reference Figure 5 As shown in (a) of [reference], taking the text extraction of itinerary information as an example, the relevant text in the original layout, such as the start and end times, location content, etc., is scattered after pasting, and the seat class information is interspersed among them.

[0177] Reference Figure 5 As shown in (b) of [reference], taking the text extraction of weather information as an example, the text in the same row in the original layout is cross-sorted after pasting, and the corresponding relationship is chaotic.

[0178] The reason is that the display order of the text during pasting is the same as the sorting of the recognized text by the system. If the system's analysis of the layout is incorrect and causes the text sorting to be disordered, which does not conform to the general reading order, then the pasted text is also difficult to reflect the reading order of the original layout, reducing the reading efficiency and the integrity of information dissemination, resulting in poor interaction experience.

[0179] In summary, the existing text extraction solutions generally have the problem of poor interaction experience. In view of this, the embodiments of the present application provide a human-computer interaction method, which can improve the interaction experience for text extraction of the text content on the page.

[0180] Figure 6 Shows a schematic flowchart of a human-computer interaction method provided by the embodiments of the present application. Figure 6The method 200 shown is applied to an electronic device, which may have a hardware structure as shown in Figure 1 and / or a software architecture as shown in Figure 2 . The method 200 includes steps S210 to S240.

[0181] S210, display a first interface, which includes text content.

[0182] In the embodiments of the present application, the text content in the first interface cannot be directly edited. For example, the first interface displays a picture, a web page, or a document (such as a PDF document) with text content, etc.

[0183] S220, in response to a first operation of the user, identify the text content.

[0184] Exemplarily, the first operation may be an operation of clicking a control for text recognition, or an operation of long pressing on the area where the text content is located. The first operation can trigger a character selection process.

[0185] In this step, the recognition result of the text content includes: the words in the text content, the spatial information (such as position and size) of the words in the text content. The recognition result of the text content can be selected and pasted. The processing of the text content involved in the following steps of the present application is actually achieved by processing the recognition result of the text content.

[0186] It can be understood that step S220 is used to convert the text content that cannot be directly edited into text content in an editable or searchable format (such as selectable and pasteable).

[0187] In some embodiments, the electronic device recognizes the text content in the first interface based on the optical character recognition OCR technology. Correspondingly, the recognition result of the text content is the OCR recognition result.

[0188] In some embodiments, in step S220, in response to the first operation of the user, the electronic device may also obtain the visual saliency features in the first interface, such as text color, font, box line, color block, shadow, etc.

[0189] S230, according to the result of recognizing the text content, generate a tree structure of the text content, which is used to represent the hierarchical information of the text content. The parent node in the tree structure corresponds to the first text block, and the child nodes of the parent node correspond to the second text blocks, and the second text blocks are located within the area of the first text block.

[0190] In this application, the text content is divided into multiple text blocks, and each text block corresponds to a node in the tree structure. The text block corresponding to the child node is located within the area of the text block corresponding to the parent node, and the content of the text block of the parent node includes the content of the text blocks of its child nodes.

[0191] In this application, a parent node in the tree structure includes multiple child nodes. Correspondingly, the first text block corresponding to the parent node includes multiple second text blocks.

[0192] S240, in response to the user's second operation, highlight at least one second text block according to the tree structure.

[0193] The second operation is an operation of selecting text. For example, the second operation is an operation of selecting text based on the cursor, an operation of selecting text based on smearing, or an operation of selecting text based on clicking. More specifically, the second operation can be a long-press operation, a swipe operation, or a click operation.

[0194] In some embodiments, the ways of highlighting at least one second text block may include at least one of the following: highlighting, adding a border line, adding an underline.

[0195] In the embodiments of this application, by constructing the tree structure of the text content in the first interface, the text content in the first interface can be hierarchically organized in the form of text blocks, so as to provide the correct reading order and support multiple reading orders. Correspondingly, when extracting text, the order in which the user selects and pastes text is more in line with the general reading habit, so as to meet the user's selection intention, improve the reading efficiency and the integrity of information dissemination. Moreover, the way for the user to select text based on text blocks is simple in operation and has a high selection efficiency. Therefore, the solution provided by this application can take into account the selection efficiency, the user's selection intention, and the reading efficiency, thus significantly improving the user's interaction experience.

[0196] For the sake of easy understanding, Figure 7 and Figure 8 respectively show partial schematic diagrams of the tree structure in combination with specific examples. It can be understood that Figure 7 and Figure 8 are only for illustrative purposes, and they respectively show the hierarchical structure of a subtree in the tree structure.

[0197] Refer to Figure 7, text block a can be divided into text blocks b and c. In the tree structure, text block a is the parent node of text blocks b and c, and text blocks b and c are sibling nodes. Text block b can be further divided into text blocks d and e, and text block c can be further divided into text blocks g1, g2, and g3. In the tree structure, text block b is the parent node of text blocks d and e, and text blocks d and e are sibling nodes. Text block c is the parent node of text blocks g1, g2, and g3, and text blocks g1, g2, and g3 are sibling nodes. The child nodes of text block b and the child nodes of text block c are cousin nodes. Text block d can be further divided into text blocks f1, f2, f3, f4, f5, and f6. In the tree structure, text block d is the parent node of text blocks f1, f2, f3, f4, f5, and f6, and text blocks f1, f2, f3, f4, f5, and f6 are sibling nodes. Text blocks e, f1, f2, f3, f4, f5, f6, g1, g2, and g3 all have only predecessor nodes and no successor nodes, so they are the leaf nodes of the tree structure.

[0198] Reference Figure 8 , text block a can be divided into text blocks b and c. In the tree structure, text block a is the parent node of text blocks b and c, and text blocks b and c are sibling nodes. Text block c can be further divided into text blocks d1, d2, d3, d4, and d5. Text block c is the parent node of text blocks d1, d2, d3, d4, and d5, and text blocks d1, d2, d3, d4, and d5 are sibling nodes. Text blocks d1, d2, d3, d4, and d5 can be further divided into smaller text blocks respectively. For example, text block d1 can be divided into text blocks e1, e2, e3, e4, e5, and e6, text block d2 can be divided into text blocks f1, f2, f3, f4, f5, and f6, and text block d5 can be divided into text blocks g1, g2, g3, g4, g5, and g6. Text blocks d1, d2, d3, d4, and d5 are the parent nodes of their child nodes respectively. The child nodes of text blocks d1, d2, d3, d4, and d5 are cousin nodes.

[0199] Figure 7 and Figure 8 In the example of, each parent node is a specific example of the first text block, and the child nodes under the parent node are specific examples of the second text block.

[0200] According to the illustration, the text block as a child node is located within the text block area of its parent node, and the content of the text block of the parent node contains the content of the text block of its child node. Thus, based on the tree-like hierarchical structure, when the user makes a selection, the user can select the text in units of text blocks. The user can not only select the low-level text blocks individually but also select the high-level text blocks individually, so it can support a more flexible text selection order. In addition, there is a hierarchical relationship between text blocks, which is more in line with the general reading order.

[0201] The text blocks involved in the embodiments of the present application are texts with a hierarchical structure, which can also be referred to as text structure bodies.

[0202] The following further details the content of generating the tree structure in step S230 with reference to the accompanying drawings.

[0203] Refer to Figure 9 , in some embodiments, the above step S230 may specifically include steps S231 to S233.

[0204] S221, divide the text content into text blocks according to the first information.

[0205] The first information may include at least one of the following: spatial information of the text, semantic information of the text, or visual saliency features in the first interface.

[0206] In some embodiments, the spatial information of the text may include the position and size of the text. The position of the text may include the coordinates of the text in the first interface and / or the text line index of the text in the first interface.

[0207] Exemplarily, the coordinates of the text in the first interface may be corner coordinates, boundary coordinates, or center point coordinates.

[0208] Exemplarily, the text line index of the text in the first interface is obtained by sorting all the words and phrases of the text content into lines and then sorting them from top to bottom. Through the text line index, the relative positions of two text blocks can be judged, for example, whether the two text blocks are on the same text line in the first interface.

[0209] Exemplarily, the size of the text may include the longitudinal dimension and the transverse dimension of the text.

[0210] It should be noted that the "text" involved in the present application may be a character, word, sentence, paragraph, chapter, or text line.

[0211] In some embodiments, the semantic information of the text may include at least one of the following: content features (i.e., specific information contained in the text, such as topics, entities, etc.), semantic features (i.e., the meaning and context relationship of the text, such as word meaning, sentence meaning, etc.), structural features (i.e., the layout and organization method of the text, such as paragraph division, headings, etc.), functional features (i.e., the purpose and role of the text, such as informative, entertaining, etc.), and tense features (i.e., the time state of the event described in the text, such as past, present, future, etc.).

[0212] In some embodiments, the visual saliency features in the first interface may include at least one of the following: text color, text font, color block, shadow, frame line, etc.

[0213] In the embodiments of the present application, when dividing text blocks, the entire page is used as the largest text block. First, the largest text block is split into multiple small text blocks, and then each small text block is further split into smaller text blocks until the smallest text block cannot be split, thus completing the division of text blocks. Therefore, the divided text blocks are in a parallel relationship or a nested relationship.

[0214] In the embodiments of the present application, the first information is used to determine the splitting axis when splitting a larger text block. Taking the first text block of the parent node and the second text block of its child node as an example, an illustration is given below.

[0215] As an example, if the first information includes visual saliency features in the first interface, such as background color differences, shadows, box lines, color blocks, etc., the splitting axis can be determined based on these visual saliency features.

[0216] For example, referring to Figure 10 in (a), if there is a straight line passing through the first text block B1 within the area of the first text block B1, the first text block B1 can be split along this straight line as the splitting axis to obtain multiple second text blocks B2.

[0217] For example, referring to Figure 10 in (b) or (c), if there is a box line within the area of the first text block B1, the first text block B1 can be split at the positions between adjacent box lines as the splitting axis to obtain multiple second text blocks B2.

[0218] For example, referring to Figure 10 in (d), if there is a color block within the area of the first text block B1, the first text block B1 can be split along the boundary of the color block as the splitting axis to obtain multiple second text blocks B2.

[0219] As another example, if the first information includes semantic information of the text, and this semantic information of the text can be used to determine the semantic relationship between texts, the splitting axis can be determined based on the semantic relationship between texts.

[0220] For example, referring to Figure 10 in (c), the first text block B1 includes outbound information and return information. "Outbound" and "return" are in a semantically equivalent corresponding relationship, indicating that they can be split. Therefore, the first text block B1 can be split at the position between the text related to the outbound information and the text related to the return information as the splitting axis to obtain multiple second text blocks B2.

[0221] For example, referring to Figure 10In (d), the texts on the left side of the first text block B1 all belong to time entities, which are significantly different from the text entity types on the right side, and semantically imply divisibility. Therefore, the first text block B1 can be divided with the position between the left time entity text and the right non-time entity text as the division axis to obtain multiple second text blocks B2.

[0222] For example, referring to Figure 11 In (a), the texts on the right side of the first text block B1 all belong to price entities, which are significantly different from the text entity types on the left side, and semantically imply divisibility. Therefore, the first text block B1 can be divided with the position between the right price entity text and the left price entity text as the division axis to obtain multiple second text blocks B2.

[0223] For example, referring to Figure 11 In (b), in the first text block B1, the text entity types of the texts in two text areas at similar positions are the same, presenting a semantic-space pattern with a very high similarity. Then, the first text block B1 can be divided with the position between the two similar layouts as the division axis to obtain multiple second text blocks B2.

[0224] As another example, the first information includes the spatial information of the text, and the spatial information of the text can be used to determine the horizontal alignment degree (or called the horizontal alignment degree) of the text, the vertical alignment degree (or called the vertical alignment degree) of the text, or the arrangement density of the text (such as the arrangement density of the text in the horizontal direction or the arrangement density of the text in the vertical direction). The horizontal alignment degree of the text, the vertical alignment degree of the text, or the arrangement density of the text can be used to determine the division axis. The arrangement density of the text can be determined according to the spacing between adjacent words in the text.

[0225] For example, referring to Figure 12 In (a) and (b), there is a text area with a relatively high vertical alignment degree (such as left alignment, right alignment, or center alignment, etc.) in the first text block B1. Then, the boundary of the text area with a relatively high vertical alignment degree can be used as the division axis to divide the first text block B1 to obtain multiple second text blocks B2. As Figure 12 shown in (a), the text area on the left side of the division axis is right-aligned vertically, then the right boundary of this text area can be used as the division axis. As Figure 12 shown in (b), the text area on the right side of the division axis is left-aligned vertically, then the left boundary of this text area can be used as the division axis.

[0226] For example, referring to Figure 12 In (c), there is a relatively large interval formed by two text areas in the first text block B1. Then, the position of this interval can be used as the division axis to divide the first text block B1 to obtain multiple second text blocks B2.

[0227] For example, referring to Figure 13 (a) therein, the adjacency matrix of the first text block B1 can be obtained first. The adjacency matrix includes the vertical distances (or includes the horizontal distances) between adjacent words in the first text block. In some embodiments, the elements in the adjacency matrix are also called spacing values, which are text spacings, such as line spacings (or column spacings). In this embodiment, the smaller the spacing value, the greater the density, and the larger the spacing value, the smaller the density. By calculating the change trend of the vertical (or horizontal) distances between adjacent words in this adjacency matrix, the values with a significant sudden increase in the vertical distance (or horizontal distance) can be found, and these values can be marked as density drop points (i.e., the positions where the text spacing increases). There may be multiple density drop points in the first text block B1, such as 16, 38, and 108 shown in the figure. Then, when splitting the first text block B1, first perform a round of DBSCAN clustering on all the words in the first text block B1 with the density drop point having the largest spacing value (such as 108). The DBSCAN clustering will cluster the words with an adjacent density greater than the density corresponding to the density drop point together, and divide the words with an adjacent density less than the density corresponding to the density drop point into different clusters, that is, cluster the words with a vertical (or horizontal) distance between adjacent words less than the spacing value corresponding to the density drop point into one text block, and cluster the words with a vertical (or horizontal) distance between adjacent words greater than the spacing value corresponding to the density drop point into different text blocks.

[0228] As Figure 13 (b) shown therein, based on the DBSCAN clustering, the first text block B1 can be divided into multiple second text blocks B2 according to the maximum value of the density drop points (such as 108), where the position of the density drop point is the position of the splitting axis.

[0229] The DBSCAN clustering algorithm is an AI algorithm model that clusters points based on the distances between points. There are two important parameters in this model that determine the performance of the clustering result: the distance matrix and the maximum neighbor distance value (eps). In this example, the distance matrix passed into the algorithm model is the above-mentioned adjacency matrix between words, and eps is the value of the located density drop point.

[0230] As mentioned above, there may be multiple density drop points in the adjacency matrix. After the first text block B1 is split into multiple second text blocks B2 through a round of DBSCAN clustering, DBSCAN clustering can be continued on the second text blocks B2 based on the second largest value of the density drop points (such as 38). For example Figure 13 (c) shown therein, based on the DBSCAN clustering, each second text block B2 can be divided into multiple third text blocks B3 according to the second largest value of the density drop points (such as 38), where the position of the density drop point is the position of the splitting axis.

[0231] That is to say, the segmentation axis of multiple hierarchical text blocks can be determined through the adjacency matrix of the first text block B1 for multi-round segmentation. Specifically, the adjacency matrix of the first text block B1 may have zero to multiple density drop points. For all density drop points, in the decreasing order of the spacing values, all words in the text block can be subjected to a round of DBSCAN clustering based on each density drop point. In each round of DBSCAN clustering, the algorithm can cluster words with an adjacent density greater than the density corresponding to the density drop point together to form new text blocks. Therefore, after each round of clustering, the original text block will be divided into several smaller text blocks. With multiple rounds of DBSCAN clustering based on density drop points with smaller values, the text blocks obtained from the previous iteration are divided into smaller text blocks until all density drop points are clustered.

[0232] In some embodiments, if the ratio of the value of a certain density drop point in the adjacency matrix to the size (such as the vertical size) of the smallest character in the first text block B1 is less than a preset threshold, the text block will no longer be segmented based on the value of this density drop point. This is because the smaller the value of the density drop point, the higher the density of the word arrangement and the smaller the distance between adjacent words. When the ratio of the distance between adjacent words to the size of the smallest character in the first text block B1 is less than the preset threshold, it can be considered that the adjacent words have an associated relationship and should not be segmented into different text blocks.

[0233] In some embodiments, if the distances between adjacent words in the first text block B1 are all equal, there are zero density drop points in the first text block B1, but a round of DBSCAN clustering can be performed on the first text block B1 based on the maximum value (all are the maximum values) in the adjacency matrix.

[0234] In the embodiments of the present application, in order to divide a larger text block into smaller text blocks, the segmentation axis determined according to the first information needs to penetrate the text block to be segmented. For example, the segmentation axis for dividing the first text block B1 into multiple second text blocks B2 needs to penetrate the first text block B1.

[0235] In some embodiments, the horizontal alignment degree of the text involved in the present application can be determined according to at least one of the differences in the character size between characters, the differences in the upper boundaries between characters, the differences in the lower boundaries between characters, and the differences in the vertical coordinates of the central points between characters in the text.

[0236] In some embodiments, the vertical alignment degree of the text can be determined according to at least one of the differences in the character size between characters, the differences in the left boundaries between characters, the differences in the right boundaries between characters, and the differences in the horizontal coordinates of the central points between characters in the text.

[0237] For ease of understanding, refer to Figure 14As shown in (a) therein, a character has a border that can enclose it, and this border is composed of an upper border, a lower border, a left border, and a right border. The center point of this border is the center point of the character. Refer to Figure 14 As shown in (b) therein, the horizontal alignment degree of the text is used to describe the differences in the vertical direction of each character in a line of text. For example, the horizontal alignment degree of the text can be determined by the differences in vertical dimensions, upper border differences, lower border differences, and the vertical coordinate differences of the center points between adjacent characters, etc. Refer to Figure 14 As shown in (c) therein, the vertical alignment degree of the text is used to describe the differences in the horizontal direction of each character in a column of text. For example, the vertical alignment degree of the text can be determined by the differences in horizontal dimensions, left border differences, right border differences, and the horizontal coordinate differences of the center points between adjacent characters.

[0238] Exemplarily, the horizontal alignment degree of the text can be quantified by formula (1) as follows:

[0239]

[0240] In formula (1):

[0241] Rx represents the horizontal alignment degree of a line of text. The larger Rx is, the higher the horizontal alignment degree of this line of text;

[0242] y i represents the vertical coordinate of the i-th character in a line of text;

[0243] y i-1 represents the vertical coordinate of the (i - 1)-th character in a line of text;

[0244] h i represents the vertical dimension of the i-th character in a line of text;

[0245] h i-1 represents the vertical dimension of the (i - 1)-th character in a line of text;

[0246] n represents the number of characters included in a line of text.

[0247] In formula (1), the horizontal alignment degree of a line of text can be calculated by the mean value of the differences between adjacent characters in this line of text.

[0248] Exemplarily, the vertical alignment degree of the text can be quantified by formula (2) as follows:

[0249]

[0250] In formula (2):

[0251] Ry represents the vertical alignment degree of a column of text. The larger Ry is, the higher the vertical alignment degree of the column of text is;

[0252] L i represents the abscissa of the left boundary of the i-th character in a column of text;

[0253] L i-1 represents the abscissa of the left boundary of the (i - 1)-th character in a column of text;

[0254] R i represents the abscissa of the right boundary of the i-th character in a column of text;

[0255] R i-1 represents the abscissa of the right boundary of the (i - 1)-th character in a column of text;

[0256] C i represents the abscissa of the center point of the i-th character in a column of text;

[0257] C i-1 represents the abscissa of the center point of the (i - 1)-th character in a column of text;

[0258] h i represents the vertical dimension of the i-th character in a column of text;

[0259] h i-1 represents the vertical dimension of the (i - 1)-th character in a column of text;

[0260] m represents the number of characters included in a column of text.

[0261] In formula (2), the vertical alignment degree of a column of text can be calculated through the average value of the differences between adjacent characters in a column of text.

[0262] In some embodiments, the splitting axis determined according to the first information can be used to determine the typesetting direction of the text block.

[0263] In the embodiments of the present application, the typesetting direction of the text block is consistent with the general reading order. A horizontally typeset text block is defined as a text block whose general reading order is to read line by line from left to right first and then wrap lines from top to bottom. A vertically typeset text block is defined as a text block whose general reading order is to read column by column from top to bottom first and then change columns from left to right. Horizontally typeset text blocks and vertically typeset text blocks may be nested, that is, smaller vertically typeset text blocks may appear within a horizontally typeset text block. Similarly, smaller horizontally typeset text blocks may appear within a vertically typeset text block. Of course, due to different splitting granularities, smaller horizontally typeset text blocks may also appear within a horizontally typeset text block. Similarly, smaller vertically typeset text blocks may appear within a vertically typeset text block.

[0264] For ease of understanding, the following takes the first text block of the parent node and the second text block of its child node as examples for illustration.

[0265] Refer to Figure 15 In (a) of, the general reading order of the first text block B1 within the solid-line frame in the figure is to read row by row from left to right first, and then read line by line from top to bottom. Therefore, the layout direction of the first text block B1 within the solid-line frame is horizontal layout.

[0266] Refer to Figure 15 In (b) of, the general reading order of the first text block B1 within the solid-line frame in the figure is to read row by row from left to right first, and then read line by line from top to bottom. Therefore, the layout direction of the first text block B1 within the solid-line frame is horizontal layout. The general reading order of the second text block B2 is to read column by column from top to bottom first, and then read column by column from left to right. Therefore, the layout direction of the second text block B2 is vertical layout. That is to say, the second text block B2 with vertical layout is nested in the first text block B1 with horizontal layout.

[0267] Refer to Figure 15 In (c) of, the general reading order of the first text block B1 within the solid-line frame in the figure is to read column by column from top to bottom first, and then read column by column from left to right. Therefore, the layout direction of the first text block B1 within the solid-line frame is vertical layout.

[0268] Refer to Figure 15 In (d) of, the general reading order of the first text block B1 within the solid-line frame in the figure is to read column by column from top to bottom first, and then read column by column from left to right. Therefore, the layout direction of the first text block B1 within the solid-line frame is vertical layout. The general reading order of the second text block B2 is to read row by row from left to right first, and then read line by line from top to bottom. Therefore, the layout direction of the second text block B2 is horizontal layout. That is to say, the second text block B2 with horizontal layout is nested in the first text block B1 with vertical layout.

[0269] In some embodiments, if the splitting axes determined according to the first information are all horizontal splitting axes (i.e., extending horizontally and passing through the first text block) or all vertical splitting axes (i.e., extending vertically and passing through the first text block), then the first text block can be directly split.

[0270] For example, if the splitting axes determined according to the first information are all horizontal splitting axes, then it can be considered that the layout direction of the first text block is horizontal layout, so that the first text block can be split with the horizontal splitting axis.

[0271] For another example, if the splitting axes determined according to the first information are all vertical splitting axes, then it can be considered that the layout direction of the first text block is vertical layout, so that the first text block can be split with the vertical splitting axis.

[0272] In some other embodiments, if the splitting axes determined according to the first information include both horizontal splitting axes and vertical splitting axes, it is necessary to accurately detect the layout direction of the first text block to determine the splitting axes for actual splitting, so as to avoid incorrect reading order caused by inappropriate splitting. Of course, if the splitting axes determined according to the first information are all horizontal splitting axes or all vertical splitting axes, it is also possible to accurately detect the layout direction of the first text block to filter out inappropriate splitting axes and avoid incorrect reading order caused by inappropriate splitting.

[0273] In some embodiments, taking the splitting of the first text block as an example, step S231 may specifically include:

[0274] Obtain the first text block;

[0275] Determine at least one candidate splitting axis according to the fourth information, where the candidate splitting axis penetrates the first text block;

[0276] Determine the layout direction of the first text block according to at least one candidate splitting axis, where the layout direction of the first text block is horizontal layout or vertical layout;

[0277] Split the first text block based on the first splitting axis among at least one candidate splitting axis to obtain a plurality of second text blocks, and the extending direction of the first splitting axis is the same as the layout direction of the first text block.

[0278] That is to say, when splitting the first text block, first detect at least one candidate splitting axis existing in the first text block, determine the layout direction of the first text block according to the at least one candidate splitting axis, and then determine the first splitting axis for actual splitting from the at least one candidate splitting axis based on the layout direction of the first text block to split the first text block. This can make the split text blocks more conform to the general reading order and avoid incorrect sorting of the text blocks.

[0279] It should be noted that the extending direction of the above-mentioned first splitting axis is the same as the layout direction of the first text block, which can be understood as follows: if the layout direction of the first text block is horizontal layout, the first splitting axis extends horizontally; if the layout direction of the first text block is vertical layout, the first splitting axis extends vertically.

[0280] In some embodiments, the layout direction of the second text block is horizontal layout or vertical layout.

[0281] In this embodiment, the method for determining at least one candidate splitting axis according to the fourth information may be the same as the method for determining the splitting axis according to the first information described above. The difference is that the candidate splitting axes determined here may not all be used to split the first text block, and the splitting axis actually used to split the first text block is selected from at least one candidate splitting axis. For detailed content, reference can be made toFigures 10 to 13 For the related description, for the sake of brevity, only a brief description is given here.

[0282] In some embodiments, the fourth information is the same as the first information. Exemplarily, the fourth information includes at least one of the following information: the visual salience features in the first interface (refer to Figure 10 ), the horizontal alignment degree of the text (refer to Figure 12 ), the vertical alignment degree of the text (refer to Figure 12 ), the mapping relationship between the semantic information of the text and the spatial information of the text (refer to Figure 11 ), and the layout density of the text (refer to Figure 13 ).

[0283] In some embodiments, the fourth information includes the layout density of the text. The candidate segmentation axis can be determined through the layout density of the text, and the typesetting direction of the first text block can be determined. Specifically, the segmentation process of the first text block in step S231 can be as follows:

[0284] Obtain the adjacency matrix of the first text block, where the adjacency matrix includes the distances between adjacent words in the first text block in the vertical or horizontal direction;

[0285] Determine the position corresponding to the maximum value in the adjacency matrix in the first text block as the candidate segmentation axis;

[0286] Determine the typesetting direction of the first text block according to at least one candidate segmentation axis;

[0287] Determine the first segmentation axis for splitting the first text block according to the typesetting direction of the first text block;

[0288] Cluster the words on both sides of the first segmentation axis to obtain a plurality of second text blocks, where the distances between adjacent words in the second text block in the vertical or horizontal direction are less than the maximum value in the adjacency matrix.

[0289] Regarding the specific content of splitting the first text block based on the adjacency matrix, reference can be made to Figure 13 For the related description, for the sake of brevity, it will not be elaborated here.

[0290] In some embodiments, the splitting of the second text block may also be included in step S231. The specific process can be as follows:

[0291] Determine the position corresponding to the second largest value in the adjacency matrix in the second text block as the second segmentation axis, and the second segmentation axis penetrates the second text block;

[0292] Split the second text block based on the second segmentation axis to obtain a plurality of third text blocks, where the distances between adjacent words in the third text block in the vertical or horizontal direction are less than the second largest value in the adjacency matrix.

[0293] Regarding the content of multi-round segmentation based on the adjacency matrix, reference can be made to Figure 13 the relevant description in, and for the sake of brevity, it will not be elaborated here.

[0294] In some embodiments, the ratio of the maximum value to the vertical height of the smallest word in the first text block is greater than or equal to a preset threshold. Exemplarily, the preset threshold is greater than 0.

[0295] In some embodiments, while determining at least one candidate segmentation axis, the candidate segmentation axis can be marked or located within the first text block.

[0296] The method for determining the layout direction according to at least one candidate segmentation axis (or potential segmentation axis) provided by the present application will be described in detail below with specific examples.

[0297] Figure 16 shows a schematic diagram of a page related to weather information, where Figure 16 in (a) shows one or more vertical candidate segmentation axes detected for the first text block B1, Figure 16 and in (b) shows one or more horizontal candidate segmentation axes detected for the first text block B1.

[0298] Figure 17 shows a schematic diagram of a page related to itinerary information, where Figure 17 in (a) shows one or more horizontal candidate segmentation axes detected for the first text block B1, Figure 17 and in (b) shows one or more vertical candidate segmentation axes detected for the first text block B1.

[0299] Figure 18 shows a schematic diagram of a tabular layout, where Figure 18 in (a) shows one or more horizontal candidate segmentation axes detected for the first text block B1, Figure 18 and in (b) shows one or more vertical candidate segmentation axes detected for the first text block B1.

[0300] It should be noted that the positions of the segmentation axes shown in the drawings of the embodiments of the present application are only schematic. In addition, the area range of the text block is also only schematic and should not be construed as a limitation to the present application.

[0301] In the embodiments of the present application, the process of determining the layout direction of the first text block according to at least one candidate segmentation axis is as follows:

[0302] S1, based on one or more detected horizontal candidate segmentation axes, pre-segment the first text block B1 to obtain multiple line text areas.

[0303] For example Figure 16 As shown in (b) of, after pre-segmenting the first text block B1 based on multiple horizontal candidate segmentation axes, line text areas C1, C2, C3, C4, C5, C6, and C7 can be obtained. For example Figure 17 as shown in (a) of Figure 18 and (a) of, after pre-segmenting the first text block B1 based on multiple horizontal candidate segmentation axes, line text areas C1, C2, C3, C4, C5, and C6 can be obtained.

[0304] S2. Determine the inter-line layout similarity of the first text block B1 according to the fifth information.

[0305] The fifth information may include at least one of the alignment method (or alignment degree) difference between line text intervals, the line width difference between line text intervals, the text number difference between line text intervals, the same-column text size difference between line text intervals, and the line spacing difference between line text intervals.

[0306] Exemplarily, the alignment method of the line text area refers to the alignment method of the text in the line text area, such as left alignment, right alignment, or center alignment, etc. The line width of the line text area refers to the horizontal dimension of the line text area. The text number of the line text area may refer to the number of words and phrases included in the line text area. The same-column text between line text intervals refers to the text corresponding to each other longitudinally in different line text areas.

[0307] In the embodiments of the present application, according to the fifth information, the layout similarity between any two line text areas can be obtained. Exemplarily, in S2, the inter-line layout similarity of the first text block B1 can be determined according to the layout similarity between adjacent line text areas.

[0308] Taking Figure 16 (b) of as an example, the inter-line layout similarity of the first text block B1 can be determined according to the layout similarity between line text area C1 and C2, the layout similarity between line text area C2 and C3, the layout similarity between line text area C3 and C4, the layout similarity between line text area C4 and C5, the layout similarity between line text area C5 and C6, and the layout similarity between line text area C6 and C7.

[0309] Exemplarily, if each line text area includes one line of text, the inter-line layout similarity of the first text block B1 can be quantified by formula (3) as follows:

[0310]

[0311] In formula (3):

[0312] h i,j represents the vertical dimension of the character in the i-th row and j-th column;

[0313] hi,j-1 Represents the vertical dimension of the character in the i-th row and (j - 1)-th column;

[0314] T i,j+1 Represents the ordinate of the upper boundary of the character in the i-th row and (j + 1)-th column;

[0315] B i,j Represents the ordinate of the lower boundary of the character in the i-th row and j-th column;

[0316] T i,j Represents the ordinate of the upper boundary of the character in the i-th row and j-th column;

[0317] B i,j-1 Represents the ordinate of the lower boundary of the character in the i-th row and (j - 1)-th column;

[0318] m represents the number of rows;

[0319] n represents the number of columns.

[0320] In formula (3), the inter-line layout similarity of the first text block B1 can be calculated through the mean of the differences between adjacent line text areas, where the differences between adjacent line text areas are obtained by calculating the differences between the texts in the same column of adjacent line text areas.

[0321] Exemplarily, referring to Figure 17 in (a), if each line text area includes multiple lines of text, the inter-line layout similarity of the first text block B1 can be quantified by formula (4) as follows:

[0322]

[0323] In formula (4):

[0324] h q,i,j Represents the vertical dimension of the character in the i-th row and j-th column of the q-th line text area;

[0325] h q,i,j-1 Represents the vertical dimension of the character in the i-th row and (j - 1)-th column of the q-th line text area;

[0326] T q,i,j+1 Represents the ordinate of the upper boundary of the character in the i-th row and (j + 1)-th column of the q-th line text area;

[0327] B q,i,j Represents the ordinate of the lower boundary of the character in the i-th row and j-th column of the q-th line text area;

[0328] T q,i,j Represents the ordinate of the upper boundary of the character in the i-th row and j-th column of the q-th line text area;

[0329] B q,i,j-1The vertical coordinate of the lower boundary of the character at the \(i\)-th row and \((j - 1)\)-th column in the \(q\)-th line text area;

[0330] \(m\) represents the number of line text areas;

[0331] \(n\) represents the number of columns in a line text area (i.e., the number of columns in a line text area);

[0332] \(p\) represents the number of rows in a line text area (i.e., the number of rows in a line text area).

[0333] In formula (4), the inter-line layout similarity of the first text block B1 can be calculated through the average of the differences between adjacent line text areas, where the differences between adjacent line text areas are obtained by calculating the differences between the corresponding line texts in adjacent line text areas. Among them, the differences between the corresponding line texts in adjacent line text areas include: the difference between the first line text of one line text area and the first line text of another line text area, the difference between the second line text of one line text area and the second line text of another line text area, and so on.

[0334] S3. Based on one or more detected vertical candidate segmentation axes, pre-segment the first text block B1 to obtain multiple column text areas.

[0335] For example Figure 16 As shown in (a) in, after pre-segmenting the first text block B1 based on multiple vertical candidate segmentation axes, column text areas P1, P2, P3, P4, and P5 can be obtained. For example Figure 17 As shown in (b) in or Figure 18 As shown in (b) in, after pre-segmenting the first text block B1 based on multiple vertical candidate segmentation axes, column text areas P1, P2, P3, and P4 can be obtained.

[0336] S4. Determine the inter-column layout similarity of the first text block B1 according to the sixth information.

[0337] The sixth information includes at least one of the alignment method difference between column text areas, the column width difference between column text areas, the text number difference between column text areas, the same-row text size difference between column text areas, and the column spacing difference between column text areas.

[0338] Exemplarily, the alignment method of a column text area refers to the alignment method of the text in the column text area, such as top alignment, bottom alignment, or middle alignment, etc. The column width of a column text area refers to the horizontal dimension of the column text area. The text number of a column text area can refer to the number of words and phrases included in the column text area. The same-row text between column text areas refers to the texts corresponding to each other horizontally in different column text areas.

[0339] In the embodiments of the present application, according to the sixth information, the layout similarity between any two column text areas can be obtained. Exemplarily, in S4, the inter-column layout similarity of the first text block B1 can be determined according to the layout similarity between adjacent column text areas.

[0340] Taking Figure 16 (a) in as an example, the inter-column layout similarity of the first text block B1 can be determined according to the layout similarity between the column text areas P1 and P2, the layout similarity between the column text areas P2 and P3, the layout similarity between the column text areas P3 and P4, and the layout similarity between the column text areas P4 and P5.

[0341] Exemplarily, if each column text area includes a column of text, the inter-column layout similarity of the first text block B1 can be quantified by formula (5) as follows:

[0342]

[0343] In formula (5):

[0344] h i,j represents the vertical dimension of the character in the i-th row and j-th column;

[0345] h i,j-1 represents the vertical dimension of the character in the i-th row and (j - 1)-th column;

[0346] L i,j+1 represents the abscissa of the left boundary of the character in the i-th row and (j + 1)-th column;

[0347] R i,j represents the abscissa of the right boundary of the character in the i-th row and j-th column;

[0348] L i,j represents the abscissa of the left boundary of the character in the i-th row and j-th column;

[0349] R i,j-1 represents the abscissa of the right boundary of the character in the i-th row and (j - 1)-th column;

[0350] m represents the number of rows;

[0351] n represents the number of columns.

[0352] In formula (5), the inter-column layout similarity of the first text block B1 can be calculated through the mean value of the differences between adjacent column text areas, where the differences between adjacent column text areas are obtained by calculating the differences between the same-row texts in adjacent column text areas.

[0353] Exemplarily, if each column text area includes multiple columns of text, the inter-column layout similarity of the first text block B1 can be quantified by formula (6) as follows:

[0354]

[0355] In formula (6):

[0356] h q,i,j represents the vertical dimension of the character at the i-th row and j-th column in the q-th column text area;

[0357] h q,i,j-1 represents the vertical dimension of the character at the i-th row and (j - 1)-th column in the q-th column text area;

[0358] L i,j+1 represents the abscissa of the left boundary of the character at the i-th row and (j + 1)-th column in the q-th column text area;

[0359] R q,i,j represents the abscissa of the right boundary of the character at the i-th row and j-th column in the q-th column text area;

[0360] L q,i,j represents the abscissa of the left boundary of the character at the i-th row and j-th column in the q-th column text area;

[0361] R q,i,j-1 represents the abscissa of the right boundary of the character at the i-th row and (j - 1)-th column in the q-th column text area;

[0362] m represents the number of rows in a column text area (i.e., the number of rows in a column text area);

[0363] n represents the number of column text areas;

[0364] k represents the number of columns in a column text area (i.e., the number of columns in a column text area).

[0365] In formula (6), the inter-column layout similarity of the first text block B1 is calculated through the mean value of the differences between adjacent column text areas, where the differences between adjacent column text areas are obtained by calculating the differences between the corresponding column texts in adjacent column text areas. Among them, the differences between the corresponding column texts in adjacent column text areas include: the difference between the first column text of one column text area and the first column text of another column text area, the difference between the second column text of one column text area and the second column text of another column text area, and so on.

[0366] It should be noted that the aforementioned formulas (1) to (6) are independent formulas respectively, and the meanings of the parameters included in each formula are only defined for each formula separately. The parameter symbols and definitions in a certain formula do not impose restrictions on other formulas.

[0367] S5. Determine the typesetting direction of the first text block B1 according to the inter-line layout similarity and the inter-column layout similarity of the first text block B1.

[0368] Specifically, when the layout similarity between lines of the first text block B1 is greater than the layout similarity between columns of the first text block B1, such as Sx is greater than Sy, the typesetting direction of the first text block is determined to be horizontal typesetting. When the layout similarity between columns of the first text block B1 is greater than the layout similarity between lines of the first text block B1, such as Sy is greater than Sx, the typesetting direction of the first text block is determined to be vertical typesetting.

[0369] For example, according to Figure 16 As can be seen from (a) in the figure, the differences between the text areas in each column are small in terms of alignment, column width, number of texts, size of the same text, and column spacing. Figure 16 As can be seen from (b), at least the text areas C5 and C6 are quite different from other text areas in terms of alignment, text size, and line spacing. Therefore, the layout similarity between columns of the first text block B1 is greater than the layout similarity between lines, and the first text block B1 is vertically laid out.

[0370] For example, according to Figure 17 As can be seen from (a) in the figure, the differences in alignment, line width, number of texts, size of co-located texts, and line spacing between the text areas are relatively small. Figure 17 As can be seen from (b), at least the column text area P4 is quite different from other column text areas in terms of column width and number of texts. Therefore, the inter-row layout similarity of the first text block B1 is greater than the inter-column layout similarity, and the first text block B1 is horizontally laid out.

[0371] For example, according to Figure 18 As can be seen from (a) in the figure, the differences in alignment, line width, number of texts, size of co-located texts, and line spacing between the text areas are relatively small. Figure 18 As can be seen from (b) in FIG. 1 , each column text area has a large difference in at least column width. Therefore, the layout similarity between lines of the first text block B1 is greater than the layout similarity between columns, and the first text block B1 is horizontally arranged.

[0372] In the above embodiment, at least one candidate segmentation axis includes one or more horizontal candidate segmentation axes and one or more vertical candidate segmentation axes, so the inter-row layout similarity and inter-column layout similarity of the first text block B1 can be obtained through the above steps S1 to S5.

[0373] In some other embodiments, if at least one candidate segmentation axis is a horizontal candidate segmentation axis, the inter-line layout similarity of the first text block can be obtained based on the above steps S1 and S2, and when the inter-line layout similarity of the first text block is greater than a first preset value, it is determined that the layout direction of the first text block is horizontal layout. Otherwise, it is considered that the first text block should not be further segmented.

[0374] In some other embodiments, if all of the at least one candidate splitting axis are vertical candidate splitting axes, the inter-column layout similarity of the first text block may be obtained based on the above steps S3 and S4. When the inter-column layout similarity of the first text block is greater than a second preset value, it is determined that the layout direction of the first text block is vertical layout. Otherwise, it is considered that the first text block is not suitable for further splitting.

[0375] In some other embodiments, if no vertical splitting axis passing through the first text block is detected, the layout direction of the first text block may be defaulted to horizontal layout.

[0376] In some embodiments, after determining the layout direction of the first text block, the electronic device may mark its layout direction for sorting text blocks at the same level when constructing a tree structure.

[0377] In some embodiments, after determining the layout direction of the first text block B1, the first splitting axis may be determined from at least one candidate splitting axis according to the layout direction of the first text block B1.

[0378] Specifically, when the layout direction of the first text block B1 is horizontal layout, the first splitting axis is the aforementioned horizontal candidate splitting axis. When the layout direction of the first text block B1 is vertical layout, the first splitting axis is the aforementioned vertical candidate splitting axis.

[0379] Based on the selected splitting axis, the first text block may be split into smaller text blocks to achieve a multi-column effect.

[0380] For example, referring to Figure 16 , the first text block B1 is in vertical layout. Therefore, the vertical candidate splitting axis is the splitting axis for actually splitting the first text block B1, that is, the first splitting axis. Splitting the first text block B1 based on the selected first splitting axis can obtain smaller second text blocks, such as the text blocks formed by the column text areas P1, P2, P3, and P4 respectively.

[0381] For example, referring to Figure 17 and Figure 18 , the first text block B1 is in horizontal layout. Therefore, the horizontal candidate splitting axis is the splitting axis for actually splitting the first text block B1, that is, the first splitting axis. Splitting the first text block B1 based on the selected first splitting axis can obtain smaller second text blocks, such as the text blocks formed by the row text areas C1, C2, C3, C4, C5, and C6 respectively.

[0382] Using the layout direction of the first text block to screen out the splitting axis actually used for splitting from at least one candidate splitting axis can eliminate inappropriate splitting axes and avoid unreasonable splitting methods. For example Figure 18For the tabular horizontal alignment layout shown, although a candidate splitting axis formed by strict left - right alignment can be found, and a relatively large horizontal gap formed by the text on both sides can also be found, the tabular layout cannot be simply split vertically by columns, otherwise it will violate the general reading order. By using the layout direction of the first text block to screen the candidate splitting axis, the inter - column splitting axis in the tabular layout can be eliminated to ensure that the tabular layout is split according to the general reading order.

[0383] In some embodiments, for the layout of tabular typesetting, the inter - column splitting axis can also be eliminated in the following way. That is, if it is determined that the horizontal alignment degree of the column text areas on both sides of the vertical candidate splitting axis is relatively high (such as greater than threshold A), and the change in the text size within each column text area is relatively small (such as less than threshold B), then it can be determined that the vertical candidate splitting axis is suspected to be the inter - column splitting axis of the table. When determining the first splitting axis, this vertical candidate splitting axis is eliminated.

[0384] In the above embodiments, the first splitting axis used to actually split the first text block is screened from the candidate splitting axes based on the layout direction of the first text block after detecting the layout direction of the first text block. In some other embodiments, the process of determining the first splitting axis and the process of detecting the layout direction of the first text block can be decoupled. For example, the layout direction of the first text block can be achieved by any of the layout direction detection methods introduced above. After determining the layout direction of the first text block, the electronic device can, based on the layout direction, use any of the above - mentioned methods for determining the splitting axis according to the first information to determine the first splitting axis. For example, if it is determined that the layout direction of the first text block is horizontal typesetting, the horizontal splitting axis used to actually split the first text block can be determined using the first information. If it is determined that the layout direction of the first text block is vertical typesetting, the vertical splitting axis used to actually split the first text block can be determined using the first information.

[0385] For example, the electronic device can first detect whether there is a vertical candidate splitting axis passing through the first text block. If not, it directly determines that the layout direction of the first text block is horizontal typesetting. Based on the horizontal layout direction of the first text block, the electronic device then determines the horizontal splitting axis for splitting the first text block according to the first information.

[0386] For another example, the electronic device can first detect whether there is a horizontal candidate splitting axis passing through the first text block. If not, it directly determines that the layout direction of the first text block is vertical typesetting. Based on the vertical layout direction of the first text block, the electronic device then determines the vertical splitting axis for splitting the first text block according to the first information.

[0387] For another example, if the electronic device detects a horizontal candidate segmentation axis and a vertical candidate segmentation axis that penetrate the first text block, and determines the layout direction of the first text block based on the horizontal candidate segmentation axis and the vertical candidate segmentation axis. Based on the layout direction of the first text block, the electronic device can further determine the segmentation axis for actually segmenting the first text block according to the first information.

[0388] In other words, in the process of detecting the layout direction of the first text block and determining the actual segmentation axis, the step of detecting the segmentation axis is performed. The methods of detecting the segmentation axis twice can be the same or different.

[0389] In the embodiments of the present application, there are various ways to obtain the first text block.

[0390] As an example, the first text block can be obtained by segmenting a larger text block. The segmentation method can refer to the segmentation method of the first text block.

[0391] As another example, the horizontal alignment degree of each line of text and / or the vertical alignment degree of each column of text in the text content can be obtained; multiple consecutive text lines and / or multiple consecutive text columns are determined as the first text block, where the horizontal alignment degree of the text line is greater than a first threshold, and the vertical alignment degree of the text column is greater than a second threshold.

[0392] For example, the text content can be scanned from beginning to end to find text lines with continuous horizontal alignment or text columns with continuous vertical alignment, and these consecutive text lines or consecutive text columns are determined as the first text block. In some embodiments, the text lines with continuous horizontal alignment can be referred to as potential lines, and the text columns with continuous vertical alignment can be referred to as potential columns. A potential line is defined as a composition formed by words that are continuously top-aligned, bottom-aligned, or center-aligned. A potential column is defined as a combination formed by words that are continuously left-aligned, right-aligned, or center-aligned.

[0393] The above mainly describes in detail the division of the text block in step S231. Now return to Figure 9 , and introduce steps S232 and S233.

[0394] As Figure 9 shown, after the division of the text block is completed, in step S232, according to the second information, the hierarchical relationship between the divided text blocks is determined.

[0395] The second information can include at least one of the spatial information of the text, the semantic information of the text, and the visual saliency features in the first interface. For the specific description of the second information, reference can be made to the above introduction of the first information. For the sake of brevity, it will not be elaborated here.

[0396] As an example, the second information may include the spatial information of the text. For example, if it is determined according to the spatial information of a text block that a certain text block is located within the area of another text block, it can be determined that the level of the other text block is higher than that of this text block. Another example is that if it is determined according to the spatial information of text blocks that two text blocks are on the same line of the first interface, it can be determined that the levels of the two text blocks are the same. For another example, if a text block is split into smaller text blocks, the level of the smaller text blocks is lower than that of the text block being split.

[0397] As another example, the second information may include the semantic information of the text. For example, if the semantics of two text blocks are opposite, such as "outbound journey" and "return journey", it can be determined that the levels of the two text blocks are the same.

[0398] As yet another example, the second information may include visual saliency features in the first interface, such as shadows, color blocks, frame lines, etc. For example, if two text blocks are located in different color block areas or frame line areas, it can be determined that the levels of the two text blocks are the same. Another example is that if the background color range of a certain text block is larger than that of another text block, it can be determined that the level of this text block is higher than that of the other text block.

[0399] Actually, after the foregoing step S231, the text content has been split into multiple text blocks, and there is an inclusion relationship between these text blocks. In the tree structure constructed based on the split text blocks, each text block is a node of the tree structure. Each layer of the tree structure represents a text structure, and the bottom-layer leaf nodes are the original words and phrases recognized in step S210.

[0400] Through step S232, the hierarchicalization of the text content can be achieved.

[0401] S233. Sort the divided text blocks according to the third information.

[0402] The third information includes the spatial information of the text. Exemplarily, the spatial information of the text may include the position and size of the text. The position of the text may include the coordinates of the text in the first interface and / or the text line index of the text in the first interface.

[0403] In this step, according to the spatial information of the text, the positional relationship between two text blocks can be determined, so that the text blocks can be sorted in an order that is more in line with the general reading habit.

[0404] In the embodiments of the present application, based on the hierarchical relationship between text blocks determined in step S232, the text blocks can be sorted according to the following principles:

[0405] 1) For the text blocks obtained by horizontal cutting, sort each text block from top to bottom;

[0406] 2) The text blocks obtained by longitudinal cutting are sorted from left to right for each text block.

[0407] 3) First, sort the upper structure, and then sort the lower structure.

[0408] In principle 1), the positional relationship between the text blocks obtained by horizontal cutting can be determined according to the third information. For example, the relative positional relationship of the text blocks in the vertical direction can be determined according to the coordinates of the text blocks or the text line index.

[0409] In principle 2), the positional relationship between the text blocks obtained by longitudinal cutting can be determined according to the third information. For example, the relative positional relationship of the text blocks in the horizontal direction can be determined according to the coordinates of the text blocks or the text line index.

[0410] Taking the aforementioned first text block and second text block as an example, in the case where the first text block is divided based on the horizontal division axis to obtain multiple second text blocks, the multiple second text blocks are sorted from top to bottom according to the spatial information of the multiple second text blocks. For example, referring to Figure 7 , the text block a (an example of the first text block) is horizontally divided to obtain the text block b (an example of the second text block) and the text block c (an example of the second text block). Since the text block b is above the text block c, the sorting of the text block b is before, and the sorting of the text block c is after. Another example, referring to Figure 8 , the text block d1 (an example of the first text block) is horizontally divided to obtain the text blocks e1, e2, e3, e4, e5, e6 (several examples of the second text blocks). According to the spatial information of the text blocks e1, e2, e3, e4, e5, e6, the text blocks e1, e2, e3, e4, e5, e6 are sorted from top to bottom.

[0411] It can be seen that for the text blocks obtained by horizontal cutting, the direction of sorting these text blocks from top to bottom is consistent with the typesetting direction of the divided text block, which conforms to the reading order of horizontal typesetting.

[0412] Taking the aforementioned first text block and second text block as an example, in the case where the first text block is divided based on the vertical division axis to obtain multiple second text blocks, the multiple second text blocks are sorted from left to right according to the spatial information of the multiple second text blocks. For example, referring to Figure 7 , the text block c (an example of the first text block) is vertically divided to obtain the text blocks g1, g2, and g3 (several examples of the second text blocks). The text blocks g1, g2, and g3 are sorted from left to right according to the spatial information of the text blocks g1, g2, and g3. Another example, referring to Figure 8, the text block c (an example of the first text block) is vertically divided to obtain text blocks d1, d2, d3, d4, d5 (several examples of the second text block), and the text blocks d1, d2, d3, d4, d5 are sorted from left to right according to the spatial information of the text blocks d1, d2, d3, d4, d5.

[0413] It can be seen that for the text blocks obtained by vertical division, the direction of sorting these text blocks from left to right is consistent with the layout direction of the divided text block, which conforms to the reading order of vertical layout.

[0414] It should be noted that Figure 7 or Figure 8 The serial numbers on the branches shown represent the sorting order of the child nodes under each parent node.

[0415] Taking the aforementioned first text block and second text block as examples, the sorting priority of the first text block is higher than that of the second text block, that is, the sorting of the first text block is before the sorting of the second text block. Refer to Figure 8 , the text block b is the upper structure of the text blocks d1 to d5, and its order should be prior to the lower structure, and the arranged text order is more in line with the general reading habit.

[0416] After the sorting is completed, the sorting of the text blocks at the lowest level is taken as the final result. Refer to Figure 7 or Figure 8 , if the text block a is the root node, the framed numbers marked on the text block a represent the sorting order of each leaf node within the entire text block a, that is, the arrangement order of the final words and the default order of selecting text from the first interface. It can be seen that the rearranged text order is more in line with the general reading habit.

[0417] The tree structure constructed by this application for the text content can achieve precise segmentation of the original layout text and sorting that conforms to the general reading order. In the process of constructing the tree structure, compared with the layout analysis that completely relies on image information, the layout analysis based on text position information has stronger real-time performance.

[0418] In some embodiments, before step S231, step S230 may further include: dividing all the words in the text content into multiple text lines, each text line running horizontally through the first interface; sorting the multiple text lines from top to bottom to obtain the text line index of each text line. Here, the text line index is used for text block division of the text content and / or sorting of the text blocks. The text line index is global and can also be called the global line index.

[0419] For example, based on the text line index, it can be determined whether words are on the same line or vertically adjacent, for use in dividing text blocks. Another example is that based on the text line index, the relative positional relationship between text blocks can be determined, for use in sorting text blocks.

[0420] Through the steps S231 to S233 introduced above, a tree structure can be constructed for the text content. Each layer of the tree structure represents a text block, and the bottommost leaf nodes of the tree structure are individual words or continuous lines. According to the tree structure, the user can select text in units of text blocks in the first interface, and can flexibly select text blocks at different levels, which not only improves the selection efficiency but also meets the user's selection intention.

[0421] The following will introduce in more detail the content of the text selected by the user in step S240 with reference to the accompanying drawings.

[0422] In some embodiments, step 240 specifically includes: determining the text block that the user intends to select according to the trajectory of the user's text selection and the tree structure, where the trajectory of the user's text selection passes through the area of at least one second text block, and the text block that the user intends to select includes at least one second text block; highlighting at least one second text block.

[0423] That is to say, based on the tree structure, the text block that the user intends to select can be determined according to the trajectory of the user's text selection. When the user selects text, they can perform efficient text selection in units of text structures in a manner that conforms to the general reading flow, without the problem of interspersed selection and disordered selection between different structures. When there are multiple possible arrangements of text blocks, the trajectory of the user's text selection can freely express the text selection order that conforms to their intention.

[0424] Figure 19 A schematic diagram of a page related to the itinerary is shown. For clarity, only the selected text blocks are indicated by dashed border lines in the figure, and each enclosed dashed border line is a text block.

[0425] Reference Figure 19 Based on the constructed tree structure, the user can select text based on the selection cursor (including the starting position and the ending position). Correspondingly, the electronic device can determine the text block that the user intends to select according to the trajectory of the user's dragged cursor. The constructed tree structure and the starting and ending positions of the selection cursor support the user to select text in the general reading order.

[0426] In an example, the user can select one outbound flight ticket information and multiple inbound flight ticket information in the first interface through the cursor. For example Figure 19 in (a), the user can select one outbound flight ticket information by dragging the cursor. When the user drags the cursor to the right, as Figure 19As shown in (b) therein, a return flight ticket information can be continuously selected. When the user continues to drag the cursor downward, as Figure 19 shown in (c) therein, a second return flight ticket information can be continuously selected without selecting the second outbound flight ticket information.

[0427] In another example, the user can select multiple outbound information or multiple return information in the first interface through the cursor. For example Figure 19 as shown in (a) therein, the user can select an outbound flight ticket information by dragging the cursor. When the user drags the cursor downward, as Figure 19 shown in (c) therein, one or more outbound flight ticket information can be continuously selected without selecting the return flight ticket information.

[0428] When there are multiple possible permutation orders between structures, the cursor trajectory of the user's selected text can freely express the text selection order that conforms to their intention. When the user wants to select the time and departure place information of multiple flight tickets, there is no need for multiple selections, and the selection efficiency is high.

[0429] Figure 20 A schematic diagram of a page related to the itinerary is shown. For clarity, only the selected text blocks are schematically shown by dashed border lines in the figure and the smaller text blocks within each selected text block are schematically shown by solid lines. Each enclosed dashed border line and solid line frame is a text block.

[0430] Refer to Figure 20 . Based on the constructed tree structure, the user can select text based on the smearing operation. Correspondingly, the electronic device can determine the text block that the user intends to select according to the trajectory of the user's smear.

[0431] For example, refer to Figure 20 as shown in (a) therein, the user can smear horizontally to select an outbound flight ticket information and a return flight ticket information.

[0432] For example, refer to Figure 20 as shown in (b) therein, the user can smear vertically to select multiple outbound flight ticket information. Or the user can smear vertically to select multiple return flight ticket information.

[0433] For example, refer to Figure 20 as shown in (c) therein, the user can first smear horizontally and then vertically to select an outbound flight ticket information and multiple return flight ticket information.

[0434] For example, refer to Figure 20 as shown in (d) therein, the user can select multiple outbound flight ticket information and multiple return flight ticket information through a discontinuous smearing operation.

[0435] When there are multiple possible arrangements between structures, the user's smearing path can express the corresponding relationship of the hierarchy intended to be selected. Compared with smearing text lines, smearing in units of structures supports fuzzy smearing, which improves the interaction efficiency and has a high error tolerance rate.

[0436] Figure 21 A schematic diagram of a page related to transfer transactions is shown. For clarity, only the selected text block and the upper text block of the selected text block are indicated by dashed border lines and solid border lines in the figure. Each enclosed dashed border line and solid frame line is a text block.

[0437] Reference Figure 21 , based on the constructed tree structure, the user can select text based on click operations. Correspondingly, the electronic device can determine the text block intended to be selected by the user according to the click trajectory of the user (including at least one click position).

[0438] For example, referring to Figure 21 in (a), the user can click to select the text block of the lowest-level node, that is, the text block of the leaf node of the tree structure, so as to select the balance information.

[0439] For example, referring to Figure 21 in (b), the user can click to select the text block of the layer above the lowest level (i.e., the second-to-last layer) node, so as to select a transfer record without a date.

[0440] For example, referring to Figure 21 in (c), the user can click to select the text block of the third-to-last layer node, so as to select a transfer record with a date.

[0441] For example, referring to Figure 21 in (d), the user can click to select the text block of the fourth-to-last layer node, so as to select the transfer records for the whole month.

[0442] In the method of selecting text based on click operations, the user can select the text at the clicked position by clicking on the text position, or can select the lowest-level text block corresponding to the blank position by clicking on the blank position. In this way, the user can select a line of ordinary text or the information of an item with one click, which improves the interaction efficiency.

[0443] In some embodiments, before the user makes a selection by clicking, the electronic device can display the outlines of each text block on the first interface to facilitate the user to determine the position of the text block to be selected.

[0444] In some embodiments, when the user clicks to select text, the electronic device can display the outline of the lowest-level text block corresponding to the click position to facilitate the user to confirm whether it is the text block intended to be selected.

[0445] In some embodiments, the click position of the user's character selection is used to determine the hierarchical level of the selected text block, so as to determine whether the selected text is a single word, a low-level text block, or a high-level text block with a larger scope.

[0446] Exemplarily, when the user clicks to select text, the electronic device can determine the lowest-level text block corresponding to the position clicked by the user according to the position clicked by the user and the boundary information of the text blocks corresponding to each node in the tree structure, and determine the lowest-level text block as the text block intended to be selected by the user.

[0447] In the embodiments of the present application, the boundary of the text block at least encloses the text of the corresponding text block. In some embodiments, the boundary of the text block may also include an area without text. The boundaries of the respective child nodes under the parent node do not overlap each other. Exemplarily, the boundary of the text block may be a rectangular border that encloses the corresponding text.

[0448] In some embodiments, as shown in Figure 22 When constructing the tree structure, an undirected graph can be constructed between the nodes of the same layer in the tree structure, where the directly adjacent text blocks can be connected bidirectionally. The connection relationship between the nodes in the tree structure represents the available paths for selecting text, and the trajectory of the user dragging the cursor, smearing, or clicking represents the selection of the path. In this way, the tree structure can provide the user with more selection orders and can better meet the user's selection intention.

[0449] Although in the construction of the tree structure, the text is re-typeset in accordance with the general reading order, for the same layout, the user may need to select in different orders when selecting to meet diverse information dissemination needs. Therefore, the single order obtained only by re-arranging the text order cannot meet the multiple character selection intentions of the user. In this embodiment, by constructing an undirected graph between the nodes of the same layer, it is possible to infer the text block intended to be selected by the user based on the trajectory of the user's character selection and the order between the selected texts.

[0450] As Figure 23As shown in (a) of [the figure], after the user selects the first word A in the first line of the text block Q1, if the selection cursor is dragged to the right, it is speculated that the user wants to continue selecting the second word B to the right of the first word in the first line. If the user selects the first word A in the first line and then drags the selection cursor down, it is speculated that the user wants to continue selecting the first word C in the second line below A. If the user selects A and C and then continues to drag the cursor to the right, it is speculated that the user wants to continue selecting B and D to the right of A and C (that is, the words in the first two lines of the text block Q1). When the user continues to drag the word selection cursor to the right, it is speculated that the word the user intends to select has left the parent node of the current word and intends to continue selecting the child node (word) under other parent nodes. Therefore, when it is detected that the user leaves the current parent node and intends to select words in other sibling parent nodes at the same level, record the words selected by the user under the current parent node as a template, and use this template (including the initial line order of the words in the text block, etc.) to quickly select the words under the new parent node.

[0451] In Figure 23 In (c) of [the figure], the user has selected the words in the first two lines of the text block Q1. Then record the words in the first two lines of this layer of the parent node as a selection template and apply it to the word selection logic for other sibling parent nodes at the same level in the future. Therefore, when the user drags the cursor to the right, the first two lines of the text block Q2 on the right can be directly selected, and when dragged down, the first two lines of the text block Q3 below can be continued to be selected, and so on.

[0452] From Figure 22 As can be seen from the undirected graph structure shown, implementing this free word selection method based on the user's word selection trajectory depends on an undirected graph structure, which is formed based on the previously constructed tree-level structure. In this undirected graph, two-way paths are established between the child nodes (i.e., sibling nodes) of the same parent node in the same layer of the original tree-level structure. This two-way path means that the user can select from the word corresponding to one child node to the word corresponding to its sibling node in any direction, and can also select from the words under one parent node to the words under other parent nodes that are sibling nodes of this parent node.

[0453] Therefore, in some embodiments, in order to further improve the selection efficiency, the electronic device can also determine the text block that the user intends to select in the following way: when it is detected that the user's selection area switches from a child node (such as a leaf node) to a parent node, mark the selected child node in this layer as a template; when it is detected that the user's selection area switches to the sibling node of the parent node, the child nodes (such as leaf nodes) under this sibling node can be mapped and selected according to the template.

[0454] Taking the foregoing first text block and second text block as examples, when the electronic device detects that the trajectory of the user's text selection switches from the area of the second text block to the area of the first text block, the levels and sorting information of the second text block selected in the first text block in the tree structure are marked as templates; when the electronic device detects that the trajectory of the user's text selection switches from the area of the first text block to the area of a sibling node of the first text block, the text blocks that the user intends to select are determined according to the template and include the text blocks corresponding to the child nodes under the sibling node.

[0455] That is to say, the electronic device can map and select the text blocks corresponding to the child nodes of the sibling node of a certain parent node according to the selection situation of the user for the child nodes under the parent node, that is, map and select the cousin nodes of the selected child nodes. Exemplarily, the levels of the child node and its cousin nodes are the same, and the sorting or position of the child node under its parent node is the same as the sorting and position of the selected cousin node under its parent node. The following is combined with Figure 23 and Figure 24 for further description.

[0456] Referring to Figure 23 in (a), the text blocks Q1, Q2, Q3, and Q4 are sibling nodes to each other and have similar layouts, and each text block has multiple child nodes. As Figure 23 in (b) and (c), the user can drag the cursor to select multiple child nodes under the text block Q1. As Figure 23 shown in (d), when the trajectory of the user's dragged cursor switches from the area of the text block Q1 to the area of the text block Q2, the electronic device infers that the user intends to select the child nodes in the text block Q2 that correspond to the positions of the selected child nodes in the text block Q1. At this time, the electronic device automatically selects the corresponding text blocks in the text block Q2 without the user having to select them one by one. As Figure 23 shown in (e), when the trajectory of the user's dragged cursor switches from the area of the text block Q2 to the area of the text block Q3, the electronic device infers that the user intends to select the child nodes in the text block Q3 that correspond to the positions of the selected child nodes in the text block Q1. Similarly, at this time, the electronic device automatically selects the corresponding text blocks in the text block Q3 without the user having to select them one by one.

[0457] Referring to Figure 24 in (a), the text blocks Q1 and Q2 are cousin nodes to each other, and each cousin node has multiple child nodes. As Figure 24 in (b) and (c), the user can select multiple child nodes under the text block Q1 through a smearing operation (for example, select a transfer record with a date). As Figure 24As shown in (d) of [the figure], when the user's smearing trajectory switches from the parent node area of text block Q1 to the parent node area of text block Q2, the electronic device infers that the user intends to select the child node in text block Q2 that corresponds to the position of the selected child node in text block Q1. At this time, the electronic device automatically selects the corresponding text block in text block Q2 (i.e., another transfer record with a date), without the user having to select them one by one.

[0458] In some embodiments, the electronic device highlights the mapped selected text block (i.e., the text block corresponding to the child node under this sibling node, that is, the cousin node of this child node).

[0459] In some embodiments, the visual saliency features of the mapped selected text block are different from those of the second text block. For example, the color background of the mapped selected text block is different from the background color of the second text block; or, the mapped selected text block has a flashing effect; or the mapped selected text block is displayed with a structural outline.

[0460] In some embodiments, when the mapped selected text block is pasted and displayed on the second interface, its visual saliency features on the second interface can be different from those of the second text block on the second interface to prompt the user.

[0461] In some embodiments, the second interface may include selection controls to facilitate the user to select to retain the content of the mapped selected text block or delete the content of the mapped selected text block.

[0462] The process of text selection has been introduced in detail above in conjunction with the accompanying drawings. Now return to refer to Figure 2 , in some embodiments, method 200 further includes:

[0463] S250, in response to the user's paste operation, display the content of at least one second text block on the second interface according to the tree structure.

[0464] In this step, the electronic device displays the content of at least one second text block on the second interface according to the hierarchical information and text sorting information provided by the tree structure.

[0465] It can be understood that in this step, when the user selects a text block of another parent node, this step also pastes and displays the content of the text block selected by the user on the second interface.

[0466] After the user has selected characters in the order of reading, it is often necessary to paste the text in a formatted or plain text manner. In the embodiments of the present application, the tree structure constructed for the text content can provide the user with a general reading order. Therefore, when the user pastes the selected text block content onto the second interface, the selected text block will also display relevant information in an order that conforms to the reading habit.

[0467] In one example, the text blocks pasted onto the second interface are displayed in separate lines at the smallest hierarchical level of the text blocks, that is, each leaf node of the selected content is displayed on a separate line, and the content of different leaf nodes is displayed on different lines. According to the sorting of each leaf node in the tree structure, the content can be displayed on the second interface in an order that conforms to the reading order.

[0468] For example, referring to Figure 25 shown, Figure 25 the hierarchical structure of the text block shown is the same as that of Figure 7 the text block shown. When pasting and displaying the text block content shown in Figure 25 onto the second interface, the display form can be as shown in Paste Result 1, and all leaf nodes are displayed in separate lines in the order of sorting in the Figure 7 shown hierarchical structure.

[0469] Again, for example, referring to Figure 26 shown, in accordance with the hierarchical structure of the text block shown in Figure 26 when pasting and displaying it onto the second interface, the display form can be as shown in Paste Result 1, and all leaf nodes are displayed in separate lines in the order of sorting in the Figure 26 shown hierarchical structure.

[0470] Another example is, referring to Figure 27 shown, Figure 27 the hierarchical structure of the text block shown is the same as that of Figure 8 the text block shown. When pasting and displaying the text block content shown in Figure 27 onto the second interface, the display form can be as shown in Paste Result 1, and all leaf nodes are displayed in separate lines in the order of sorting in the Figure 8 shown hierarchical structure.

[0471] In another example, when the text block content pasted onto the second interface is displayed, the child nodes with the same parent node are displayed on the same line, and the child nodes with different parent nodes are displayed on separate lines.

[0472] When people read, they generally read texts that are close in position together because these texts will appear in a small reading hot zone at the same time. Generally, child nodes with the same parent node usually have a relatively high degree of proximity in position, so text blocks with the same parent node can be grouped and displayed on the same line. Here, the degree of text proximity can be judged by the granularity of the text blocks segmented in the previous steps. Generally, the smaller the granularity of the text block, the higher the degree of proximity between texts.

[0473] In some embodiments, nodes belonging to the same low level (for example, within the last three levels) under the same parent node are displayed on the same line.

[0474] In some embodiments, different text blocks are displayed on separate lines.

[0475] For example, referring to Figure 25 and Figure 7 , according to Figure 7 the hierarchical structure shown, text blocks f1, f2, f3, f4, f5, f6 have the same parent node d, so text blocks f1, f2, f3, f4, f5, f6 can be displayed on the same line. Text blocks g1, g2, and g3 have the same parent node c, so text blocks g1, g2, and g3 can be displayed on the same line. According to Figure 7 it can be known that text blocks f1 / f2 / f3 / f4 / f5 / f6 / g1 / g2 / g3 and text block e have different parent nodes, so text block c is displayed on a separate line. Exemplarily, Figure 7 the display form of the text blocks shown in Figure 25 can be like the paste result 2 shown in

[0476] Again, for example, referring to Figure 26 , according to Figure 26 the hierarchical structure shown, text blocks d1, d2 have the same parent node b, so text blocks d1, d2 can be displayed on the same line. Text blocks e1, e2, e3, and e4 have the same parent node c, so text blocks e1, e2, e3, and e4 can be displayed on the same line. The parent nodes of text blocks d1, d2 are different from the parent nodes of text blocks e1, e2, e3, and e4, so text blocks d1, d2 and text blocks e1, e2, e3, and e4 are displayed on different lines. Exemplarily, Figure 26 the display form of the text blocks shown in Figure 26 can be like the paste result 2 shown in

[0477] Also, for example, referring to Figure 27 and Figure 8 , according to Figure 8As can be seen from the hierarchical structure shown, text blocks e1, e2, e3, e4, e5, and e6 have the same parent node d1. Therefore, text blocks e1, e2, e3, e4, e5, and e6 can be displayed on the same line. Text blocks f1, f2, f3, f4, f5, and f6 have the same parent node d2. Therefore, text blocks f1, f2, f3, f4, f5, and f6 can be displayed on the same line. According to Figure 8 it can be seen that text blocks d1 and d2 are different parent nodes. Therefore, text blocks d1 and d2 are displayed on different lines. Text block b and text blocks d1 and d2 belong to different hierarchical structure diagrams. Therefore, text block b is displayed on a separate line. Exemplarily, Figure 27 the display form of the selected text block shown in the second interface can be as Figure 27 shown in the paste result 2.

[0478] In another example, when the content of the text block pasted into the second interface is displayed, text blocks belonging to the same association relationship are displayed on the same line, and text blocks belonging to different association relationships or text blocks with no association relationship are displayed on separate lines.

[0479] When people read, they will actively search for the next text information with strong semantic relevance based on semantics. For example, when looking at the flight ticket layout information and seeing the departure time, they will immediately search for the arrival time. That is, the reading order of text with strong semantic association is similar. When pasting, putting text blocks belonging to the same association relationship together for display can improve the reading efficiency.

[0480] For example, referring to Figure 25 and Figure 7 , according to semantic analysis, text blocks f1, f2, f3, f4, f5, and f6 are information related to the itinerary, specifically showing a start-end relationship. Therefore, text blocks f1, f2, f3, f4, f5, and f6 can be displayed on the same line. Text blocks g1, g2, and g3 are all seat class information. Therefore, text blocks g1, g2, and g3 can be displayed on the same line. Text block e is information related to price, which is different from the itinerary information and seat class information types. Therefore, it is displayed on a separate line. Exemplarily, Figure 25 the display form of the selected text block shown in the second interface can be as Figure 25 shown in the paste result 2.

[0481] Again, for example, referring to Figure 26 , according to semantic analysis, text blocks d1 and d2 are information related to dates, specifically showing a subordinate relationship. Therefore, text blocks d1 and d2 can be displayed on the same line. Text blocks e1, e2, e3, and e4 are all information related to transfers. Therefore, text blocks e1, e2, e3, and e4 can be displayed on the same line. Text blocks b and c belong to different entities. Therefore, they are displayed on separate lines. Exemplarily, Figure 26The display form of the selected text block shown in the second interface can be as Figure 26 the paste result 2 shown.

[0482] For another example, referring to Figure 27 and Figure 8 , according to the hierarchical structure shown in Figure 8 , it can be known that the text blocks e1, e2, e3, e4, e5, and e6 are yesterday's weather information. Therefore, the text blocks e1, e2, e3, e4, e5, and e6 can be displayed on the same line. The text blocks f1, f2, f3, f4, f5, and f6 are today's weather information. Therefore, the text blocks f1, f2, f3, f4, f5, and f6 can be displayed on the same line. According to Figure 8 , it can be known that the text blocks d1 and d2 are weather information for different dates. Therefore, the text blocks d1 and d2 are displayed on different lines. The text block b is title information. Therefore, the text block b is displayed on a separate line. Exemplarily, Figure 27 the display form of the selected text block shown in the second interface can be as Figure 27 the paste result 2 shown.

[0483] In some embodiments, when different text blocks are displayed on the same line, the text blocks can be arranged in sequence according to the sorting order of the text blocks in the tree structure (for example, in the order from top to bottom or from left to right in the first interface), such as referring to Figure 25 the paste result 2 shown, Figure 26 the paste result 2 shown, and Figure 27 the paste result 2 shown.

[0484] In some other embodiments, when different text blocks are displayed on the same line, the electronic device can reorder the text blocks. For example, they can be arranged in sequence according to the font size of the text blocks from largest to smallest. As shown in Figure 26 , the font sizes of the text blocks e1, e4, e3, and e2 decrease in sequence. Therefore, when e1, e2, e3, and e4 are displayed on the same line, they are arranged in the order of e1, e4, e3, and e2, as in Figure 26 the paste result 4.

[0485] When people read, they usually notice the text with larger font size, bold, high color contrast, and unique font style first. Sorting the pasted text based on such features can improve the reading efficiency.

[0486] In some embodiments, when different text blocks are displayed on different lines, the text blocks can be sorted by row according to the reading order of the text blocks in the original layout or the hierarchy of the text blocks in the tree structure. For example, for a structure with a higher reading order in the original layout, its row sorting in the second interface is higher, such as Figure 25 the paste result 2 shown, Figure 26The pasting result 2 shown Figure 27 The pasting result 2 shown. For example, the text block of the upper layer structure has a higher row sorting order in the second interface than the text block of the lower layer structure in the second interface. For example Figure 27 In the pasting result 2 shown, the text block of "Multi-day Forecast" with a higher level has a higher row sorting order than the text block of weather information

[0487] People's reading order of text in list-like layouts usually presents an 'F' pattern, that is, the text at the top and front in list-like layouts has a higher reading order. In this application, sorting rows according to the position information of text lines can improve the reading efficiency

[0488] In summary, reorganizing the order and line breaks of the pasted text helps to restore the structural hierarchy and correlation relationship contained in the original layout, and improves the efficiency and integrity of information dissemination

[0489] In some embodiments, when the selected content is displayed in the second interface, there is only one association relationship per line

[0490] In some embodiments, when pasting the selected text, the electronic device adds format markers to highlight the association relationship between texts and / or the hierarchical relationship of text blocks

[0491] Adding special symbols helps to restore the structural hierarchy and correlation relationship contained in the original layout, and improves the efficiency and integrity of information dissemination

[0492] Exemplarily, the format markers include at least one of the following: parentheses, semicolons, dashes, serial numbers, bullet points, line breaks, tab stops, vertical bar symbols

[0493] For example, the parallel same-level relationship can be represented by adding a line break

[0494] For example, the cross-hierarchical relationship from the parent node to the child node can be represented by adding a line break and a tab stop (tab). For example Figure 27 In the pasting result 2 in, there is a cross-hierarchical relationship between "Multi-day Forecast" and weather information. In some other embodiments, the tab stop can also be replaced by square brackets, bullet points, serial numbers, etc

[0495] Exemplarily, the association relationship between texts includes: subordinate relationship, from-to relationship (or start-end relationship), parallel relationship, no relationship

[0496] In some embodiments, it can be defined that there is a subordinate relationship between different-level texts (such as different font size levels) that are close in the horizontal / vertical direction; or there is a subordinate relationship between different-level texts (such as different font size levels) that are aligned in the horizontal / vertical direction

[0497] For example, the subordination relationship between texts can be highlighted by adding formatting marks such as parentheses and semicolons. By way of example and not limitation, the representation of the subordination relationship can be, for example: A(B; C). Refer to Figure 25 In the paste result 3 in Figure 26 Figure 26 , the subordination relationships are: "06:10 (Shanghai Hongqiao Station)", "07:38 (Nanjing Station)". Refer to Figure 27 In the paste result 3 in

[0498] In some embodiments, the texts at the same level connected by symbols such as dashes or arrows in the original layout can be defined as the from-to relationship.

[0499] For example, the from-to relationship between texts can be highlighted by adding a dash. By way of example and not limitation, the representation of the from-to relationship can be, for example: A—B (associated relationship supplementary text). Refer to Figure 25 In the paste result 3 in Figure 27 Figure 27 , the from-to relationships are: "06:10 (Shanghai Hongqiao Station)—07:38 (Nanjing Station) (1 hour and 28 minutes; G1970)".

[0500] In some embodiments, the texts arranged at equal intervals in the same row can be defined as the parallel relationship, or the texts separated by the same symbol can be defined as the parallel relationship, or the texts of the same size level can be defined as the parallel relationship.

[0501] For example, the parallel relationship between texts can be highlighted by adding a vertical bar symbol. By way of example and not limitation, the representation of the parallel relationship can be, for example: A|B|C. Refer to Figure 25 In the paste result 3 in

[0502] For example, the texts with no direct association relationship can be highlighted by adding a semicolon. By way of example and not limitation, the representation of the non-association relationship can be, for example: A; B; C. Refer to Figure 26 In the paste result 3 in

[0503] In the embodiments of the present application, when pasting text, methods such as line aggregation (i.e., displaying on the same line), line sorting (displaying in separate lines), and adding special symbols help to restore the structural hierarchy, reading order, and association relationship. Therefore, when the selected text is displayed in the second interface, the text structure is clear (such as being divided into blocks, segments, and levels), the reading order is reasonable (including reasonable reading order between structures at the same level and reasonable inclusion order between structures at different levels), and the association relationship between texts is prominent. Users can read the pasted text in a consistent reading order, thereby improving the information dissemination efficiency in text reading.

[0504] Figure 28 FIG. shows a schematic diagram of a paste result provided by an embodiment of the present application. As Figure 28 shown, when pasting 1 to N same-level structures with relative semantics, format markers can be added to reflect the corresponding relationship between the same-level structures. For example, when the user selects a one-way text message and multiple return text messages, when pasting the text, the one-to-many relationship between the return trip information and the one-way trip information can be reflected by adding bullet points, serial numbers, or indents, etc. For example, taking a one-way information as a parent node and multiple return information as child nodes of the one-way information.

[0505] In some embodiments, when it is detected that the text selected by the user has a table header or a title, even if the user does not select the corresponding table header or title, it can be added to the pasted text in the form of "

Title

[0506] In some embodiments, as shown in (a) of Figure 29 , when the electronic device detects that the user adjusts the format of a text block in the second interface, it can automatically batch-adjust the display format of the text content at the same level, as Figure 29 shown in (b) of

[0507] Exemplarily, the electronic device can batch-adjust any one of the display formats of the text content at the same level, such as font, font size, color, shadow, indent format, underline mark, etc.

[0508] In some embodiments, the selected text information in the present application can be pasted and displayed according to the following layout principles:

[0509] 1) Line aggregation principle: The text of the same text block is displayed on the same line; the text blocks belonging to the same association relationship are displayed on the same line.

[0510] 2) Line break principle: The text that is in different text blocks and has no association in the original layout is displayed in separate lines; there is only one segment of association relationship in one line.

[0511] 3) Line sorting principle: In the original layout, text blocks that are higher and to the left are sorted earlier; upper text blocks are sorted earlier than lower text blocks; and special symbols such as tab can be added between upper and lower text blocks to indicate an inclusion relationship.

[0512] 4) In-line sorting principle: From largest to smallest (font size); from top to bottom (position); from left to right (position).

[0513] 5) Principle of highlighting association relationships: Different symbols are used to highlight the association relationships between texts. For example, start-end relationship (A——B), parallel relationship (A|B|C), subordination relationship (A(B;C)).

[0514] The following takes the first text block and the second text block as examples to briefly describe the pasting method.

[0515] In some embodiments, step S250 may specifically include: displaying the content of at least one second text block in the same row in the second interface, where at least one second text block meets the first preset condition.

[0516] The first preset condition includes at least one of the following:

[0517] At least one second text block has the same parent node;

[0518] At least one second text block has the same association relationship, and the association relationship is a subordination relationship, a parallel relationship, or a start-end relationship.

[0519] Refer to Figure 7 , text blocks f1, f2, f3, f4, f5, f6 are a specific example of at least one second text block. Or text blocks g1, g2, g3 are respectively a specific example of at least one second text block.

[0520] Refer to Figure 8 , text blocks f1, f2, f3, f4, f5, f6 are a specific example of at least one second text block. Or text blocks e1, e2, e3, e4, e5, e6 are respectively a specific example of at least one second text block.

[0521] Refer to Figure 26 , text blocks d1, d2 are a specific example of at least one second text block. Or text blocks e1, e2, e3, e4 are respectively a specific example of at least one second text block.

[0522] In some embodiments, the second text block is within the last three levels in the tree structure.

[0523] In some embodiments, when the content of at least one second text block is displayed in the same row on the second interface, the display order of the content of at least one second text block is determined according to the seventh information. The seventh information includes at least one of the following information: spatial information of at least one second text block in the first interface, semantic information of at least one second text block, or at least one of the visually significant features related to at least one second text block in the first interface. For the seventh information, reference may be made to the aforementioned in-row sorting principle and the specific embodiments of in-row display.

[0524] For example, the second text block that is located at the front and top in the first interface is displayed first in the second interface, and the second text block that is located at the back and bottom in the first interface is displayed last in the second interface.

[0525] For example, the second text blocks that are strongly semantically related are displayed in a close order in the second interface.

[0526] For example, a second text block with a special font, a larger font size, or a special text color in the first interface is displayed first in the second interface.

[0527] In some embodiments, step S250 may specifically include: displaying the content of at least one second text block and the content of at least one fourth text block on the second interface, the content of the second text block and the content of the fourth text block are located in different rows of the second interface, and the second text block and the fourth text block meet the second preset condition.

[0528] The second preset condition includes at least one of the following:

[0529] The second text block and the fourth text block have different parent nodes;

[0530] The second text block and the fourth text block have different association relationships, and the association relationships include a subordinate relationship, a parallel relationship, or a start-end relationship.

[0531] refer to Figure 7 , text blocks f1, f2, f3, f4, f5, and f6 are specific examples of at least one second text block. Text blocks g1, g2, and g3 are specific examples of at least one fourth text block.

[0532] refer to Figure 8 , text blocks f1, f2, f3, f4, f5, f6 are specific examples of at least one second text block. Text blocks e1, e2, e3, e4, e5, e6 are specific examples of at least one fourth text block.

[0533] refer to Figure 26, text blocks d1 and d2 are a specific example of at least one second text block. Text blocks e1, e2, e3, and e4 are a specific example of at least one fourth text block.

[0534] In some embodiments, the second text block and the fourth text block are at levels more than four from the bottom in the tree structure.

[0535] In some embodiments, when the contents of the second text block and the fourth text block are displayed in separate lines on the second interface, the display order of the contents of at least one second text block and at least one fourth text block is determined according to the eighth information. The eighth information includes at least one of the following: the spatial information of at least one second text block in the first interface, the hierarchical information of at least one second text block, the spatial information of at least one fourth text block in the first interface, or the hierarchical information of at least one fourth text block. Regarding the eighth information, reference may be made to the aforementioned line-by-line sorting principle and specific embodiments of line-by-line display.

[0536] For example, if the position of the second text block in the first interface is higher than the position of the fourth text block in the first interface, the display order of the second text block on the second interface is above, and the display order of the fourth text block on the second interface is below.

[0537] For example, if the level of the second text block is higher than the level of the fourth text block, the display order of the second text block on the second interface is above, and the display order of the fourth text block on the second interface is below.

[0538] In some embodiments, method 200 may further include: displaying a format marker on the second interface, where the format marker is used to identify the association relationship between at least one second text block and / or the hierarchical relationship of at least one second text block in the tree structure.

[0539] In some embodiments, method 200 may further include: detecting an operation by the user to adjust the format of the content of the second text block, and batch-adjusting the display format of the text blocks at the same level as the second text block on the second interface.

[0540] Exemplarily, the operation by the user to adjust the format of the content of the second text block includes at least one of the following: changing the text color, changing the text font, changing the text size, adding / canceling underlining, adding / canceling serial numbers, and adjusting the indentation format.

[0541] As described above in conjunction with Figures 1 to 29 , the human-computer interaction method provided in the embodiments of the present application has been described in detail. Next, the device embodiments of the present application will be described in conjunction with Figures 30 to 31 in detail. It should be understood that the description of the method embodiments corresponds to the description of the device embodiments. Therefore, for parts not described in detail, reference may be made to the previous method embodiments.

[0542] Figure 30 shows a schematic structural diagram of a device provided by an embodiment of the present application. The device 300 may be located in Figure 1 the electronic device 100 shown, or be a specific example of the electronic device 100. The device 300 is capable of executing Figure 6 each step in the method shown, and may specifically implement Figures 7 - 29 the embodiment shown. To avoid redundancy, it will not be described repeatedly.

[0543] As Figure 30 shown, the device 300 may include a display unit 310, an identification unit 320, and a generation unit 330.

[0544] The display unit 310 may be used to execute Figure 6 steps S210, S240, and S250 in the method 200 shown. The display unit 310 is used to execute the steps related to display such as displaying a window, displaying an application interface, displaying a control, and displaying a page in the foregoing method embodiment.

[0545] The identification unit 320 may be used to execute Figure 6 step S220 in the method 200 shown. The identification unit 320 is mainly used to execute the steps related to identifying text content in the foregoing method embodiment.

[0546] The generation unit 330 may be used to execute Figure 6 step S230 in the method 200 shown. The identification unit 330 is mainly used to execute the steps related to generating a tree structure in the foregoing method embodiment. For example, the identification unit 330 is specifically used to execute Figure 9 steps S231 to S233 shown.

[0547] Optionally, the device 300 may further include a storage unit, and the storage unit is used to store the program code and data of the device 300.

[0548] Figure 31 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Figure 31 The electronic device 400 shown may be Figure 1 a specific example of the electronic device 100 in

[0549] Figure 30 The electronic device 400 shown includes a memory 410, a processor 420, and a bus 430. Among them, the memory 410 and the processor 420 are communicatively connected to each other through the bus 430.

[0550] The memory 410 can be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 410 can store a program. When the program stored in the memory 410 is executed by the processor 420, the processor 420 is used to execute each step of the human-computer interaction method of the embodiments of the present application.

[0551] The processor 420 can be a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits, and is used to execute relevant programs to execute the human-computer interaction method of the embodiments of the present application.

[0552] The processor 420 can also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the human-computer interaction method of the present application can be completed by the integrated logic circuit in the hardware of the processor 420 or instructions in software form. The above-mentioned processor 420 can also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit, a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read only memory, a programmable read only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 410, and the processor 420 reads the information in the memory 410 and combines its hardware to execute the human-computer interaction method of the embodiments of the present application.

[0553] In some embodiments, the electronic device 400 further includes a communication interface 440. The communication interface 440 uses a transceiver device such as, but not limited to, a transceiver to implement the communication between the electronic device 400 and other devices or communication networks.

[0554] The bus 430 may include paths for transmitting information between various components of the electronic device 400 (e.g., the memory 410, the processor 420, the communication interface 440).

[0555] This application also provides an electronic device, including: one or more processors; one or more memories; the one or more memories store one or more computer programs, and the one or more computer programs include instructions that, when executed by the one or more processors, cause the electronic device to execute each step in the method as shown in Figure 6 or implement the specific embodiments as shown in Figures 7 to 29 .

[0556] This application also provides a readable storage medium, including computer instructions that, when running on an electronic device, cause the electronic device to execute each step in the method as shown in Figure 6 or implement the specific embodiments as shown in Figures 7 to 29 .

[0557] This application also provides a chip that stores instructions that, when run on an electronic device, can implement the above human-computer interaction method.

[0558] This application also provides a computer program product that stores a program or instructions that, when run, can implement the above human-computer interaction method.

[0559] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0560] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be repeated here.

[0561] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0562] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0563] In addition, in each embodiment of the present application, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0564] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0565] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A human-computer interaction method, characterized in that: include: Display a first interface, wherein the first interface includes text content; In response to a first operation by a user, identifying the text content; Generate a first structure of the text content according to the result of identifying the text content, the first structure is used to represent hierarchical information of the text content, wherein a first node in the first structure corresponds to a first text block, a second node associated with the first node corresponds to a second text block, and the second text block is located in an area of ​​the first text block; In response to a second operation of the user, at least one of the second text blocks is displayed in a first mode according to the first structure.

2. The method according to claim 1, characterized in that The step of generating a first structure of the text content according to a result of identifying the text content comprises: Dividing the text content into text blocks according to first information, wherein the first information includes at least one of spatial information of the text, semantic information of the text, and visually significant features in the first interface; Determining a hierarchical relationship between the divided text blocks according to second information, wherein the second information includes at least one of spatial information of the text, semantic information of the text, and a visually significant feature in the first interface; The divided text blocks are sorted according to third information, where the third information includes spatial information of the text.

3. The method according to claim 2, characterized in that The dividing the text content into text blocks according to the first information includes: Get the first text block; determining at least one candidate segmentation axis according to the fourth information, wherein the candidate segmentation axis runs through the first text block; Determining a layout direction of the first text block according to at least one of the candidate segmentation axes, wherein the layout direction of the first text block is horizontal layout or vertical layout; The first text block is segmented based on a first segmentation axis among at least one of the candidate segmentation axes to obtain a plurality of the second text blocks, wherein an extension direction of the first segmentation axis is the same as a layout direction of the first text block.

4. The method according to claim 3, characterized in that: The fourth information includes at least one of the following information: visually significant features in the first interface, the horizontal alignment of the text, the vertical alignment of the text, the mapping relationship between the semantic information of the text and the spatial information of the text, and the arrangement density of the text.

5. The method according to claim 4, characterized in that The horizontal alignment of the text is determined based on at least one of the size difference between characters, the upper boundary difference between characters, the lower boundary difference between characters, and the vertical coordinate difference between the center points of the characters in the text; and / or The vertical alignment degree of the text is determined based on at least one of the size difference between characters, the left boundary difference between characters, the right boundary difference between characters, and the horizontal coordinate difference between the center points of characters in the text.

6. The method according to any one of claims 3 to 5, characterized in that At least one of the candidate segmentation axes includes one or more horizontal candidate segmentation axes and one or more vertical candidate segmentation axes, and determining the layout direction of the first text block according to the at least one candidate segmentation axis includes: Pre-segmenting the first text block based on one or more of the horizontal candidate segmentation axes to obtain multiple line text areas; Determining the layout similarity between lines of the first text block according to fifth information, the fifth information including at least one of a difference in alignment between the line text intervals, a difference in line width between the line text intervals, a difference in the number of texts in the line text intervals, a difference in the size of texts in the same column of the line text intervals, and a difference in line spacing between the line text intervals; Pre-segmenting the first text block based on one or more of the vertical candidate segmentation axes to obtain a plurality of column text areas; Determine the layout similarity between columns of the first text block according to sixth information, where the sixth information includes at least one of the alignment difference between the column text intervals, the column width difference between the column text intervals, the number of texts in the column text intervals, the size difference of the same line of text in the column text intervals, and the column spacing difference between the column text intervals; The layout direction of the first text block is determined according to the inter-row layout similarity and the inter-column layout similarity, wherein: When the layout similarity between the lines is greater than the layout similarity between the columns, the layout direction of the first text block is horizontal layout; When the layout similarity between columns is greater than the layout similarity between rows, the layout direction of the first text block is vertical layout.

7. The method according to claim 6, characterized in that Before segmenting the first text block based on a first segmentation axis among at least one of the candidate segmentation axes, the method further includes: According to the typesetting direction of the first text block, the first segmentation axis is determined from at least one of the candidate segmentation axes, wherein: When the layout direction of the first text block is horizontal layout, the first segmentation axis is the horizontal candidate segmentation axis; When the layout direction of the first text block is vertical layout, the first segmentation axis is the vertical candidate segmentation axis.

8. The method according to any one of claims 3 to 7, characterized in that The determining at least one candidate segmentation axis according to the fourth information includes: Acquire an adjacency matrix of the first text block, wherein the adjacency matrix includes vertical or horizontal distances between adjacent words in the first text block; Determine the position in the first text block corresponding to the maximum value in the adjacent matrix as the candidate segmentation axis; The step of segmenting the first text block based on a first segmentation axis among at least one of the candidate segmentation axes to obtain a plurality of second text blocks includes: Cluster the words on both sides of the first segmentation axis to obtain a plurality of the second text blocks, wherein the vertical or horizontal distance between adjacent words in the second text blocks is less than the maximum value in the adjacent matrix.

9. The method according to claim 8, characterized in that The method further comprises: determining a position in the second text block corresponding to the second maximum value in the adjacent matrix as a second segmentation axis, wherein the second segmentation axis runs through the second text block; The second text block is segmented based on the second segmentation axis to obtain a plurality of third text blocks, wherein the vertical or horizontal distance between adjacent words in the third text blocks is smaller than the second maximum value in the adjacent matrix.

10. The method according to claim 9, characterized in that The ratio of the secondary maximum value to the vertical height of the smallest character in the first text block is greater than or equal to a preset threshold.

11. The method according to any one of claims 3 to 10, characterized in that The obtaining of the first text block includes: Obtaining the horizontal alignment of each line of text and / or the vertical alignment of each column of text in the text content; A plurality of continuous text lines and / or a plurality of continuous text columns are determined as the first text block, wherein the horizontal alignment degree of the text lines is greater than a first threshold, and the vertical alignment degree of the text columns is greater than a second threshold.

12. The method according to any one of claims 2 to 11, characterized in that The spatial information of the text includes the position of the text and the size of the text, wherein the position of the text includes the coordinates of the text in the first interface and / or the text line index of the text in the first interface.

13. The method according to any one of claims 2 to 12, characterized in that The visually significant feature includes at least one of the following: text color, text font, color block, shadow, and frame line.

14. The method according to any one of claims 2 to 13, characterized in that The step of sorting the divided text blocks according to the third information includes: In the case where the first text block is divided based on the horizontal dividing axis to obtain a plurality of the second text blocks, the plurality of the second text blocks are sorted from top to bottom according to the spatial information of the plurality of the second text blocks; or In a case where the first text block is divided based on the vertical dividing axis to obtain a plurality of the second text blocks, sorting the plurality of the second text blocks from left to right according to spatial information of the plurality of the second text blocks; The sorting priority of the first text block is higher than the sorting priority of the second text block.

15. The method according to any one of claims 2 to 14, characterized in that Before dividing the text content into text blocks according to the first information, the method further includes: Divide all words in the text content into a plurality of text lines, each of which runs through the first interface in a horizontal direction; The plurality of text lines are sorted from top to bottom, and a text line index of each text line is obtained, wherein the text line index is used to divide the text content into text blocks and / or sort the text blocks.

16. The method according to any one of claims 1 to 15, characterized in that In response to the second operation of the user, highlighting at least one of the second text blocks according to the first structure includes: Determining a text block that the user intends to select according to a trajectory of the text selected by the user and the first structure, wherein the trajectory of the text selected by the user passes through an area of ​​at least one of the second text blocks, and the text block that the user intends to select includes at least one of the second text blocks; At least one of the second text blocks is highlighted.

17. The method according to claim 16, characterized in that The step of determining the text block that the user intends to select according to the trajectory of the text selected by the user and the first structure includes: detecting that the track of the text selected by the user switches from the area of ​​the second text block to the area of ​​the first text block, and marking the level and sorting information of the second text block selected in the first text block in the first structure as a template; It is detected that the trajectory of the user selecting text switches from the area of ​​the first text block to the area of ​​the brother node of the first node, and it is determined according to the template that the text block intended to be selected by the user includes the text block corresponding to the first node under the brother node.

18. The method according to claim 17, characterized in that The method further comprises: A text block corresponding to a first node under the sibling node is highlighted, wherein a visually significant feature of the text block corresponding to the first node under the sibling node is different from a visually significant feature of the second text block.

19. The method according to claim 17 or 18, characterized in that The second operation is any one of the following: an operation of selecting text based on a cursor, an operation of selecting text based on smearing, or an operation of selecting text based on clicking.

20. The method according to any one of claims 16 to 19, characterized in that The second operation is an operation based on clicking to select text, and the step of determining the text block that the user intends to select according to the trajectory of the user selecting the text and the first structure includes: Determine the lowest level text block corresponding to the position clicked by the user according to the position clicked by the user and the boundary information of the text blocks corresponding to each node in the first structure; The text block at the lowest level is determined to be the text block that the user intends to select.

21. The method according to any one of claims 1 to 20, characterized in that The method further comprises: In response to a user's paste operation, content of at least one of the second text blocks is displayed on a second interface according to the first structure.

22. The method according to claim 21, characterized in that The method of displaying the content of at least one of the second text blocks on the second interface according to the first structure in response to the user's paste operation includes: Displaying the content of at least one of the second text blocks in the second interface, wherein at least one of the second text blocks meets the first preset condition; The first preset condition includes at least one of the following: At least one of the second text blocks has a same first node; At least one of the second text blocks has the same association relationship, and the association relationship is a subordinate relationship, a parallel relationship, or a start-end relationship.

23. The method according to claim 22, characterized in that The step of displaying at least one of the second text blocks in the second interface includes: The display order of the content of at least one of the second text blocks is determined based on seventh information, and the seventh information includes at least one of the following information: spatial information of at least one of the second text blocks in the first interface, semantic information of at least one of the second text blocks, or at least one of the visually significant features related to at least one of the second text blocks in the first interface.

24. The method according to claim 21, characterized in that The method of displaying the content of at least one of the second text blocks on the second interface according to the first structure in response to the user's paste operation includes: Displaying the content of at least one of the second text blocks and the content of at least one of the fourth text blocks on the second interface, wherein the content of the second text block and the content of the fourth text block are located in different rows of the second interface, and the second text block and the fourth text block meet a second preset condition; The second preset condition includes at least one of the following: The second text block and the fourth text block have different first nodes; The second text block and the fourth text block have different association relationships, and the association relationships include a subordinate relationship, a parallel relationship, and a start-end relationship.

25. The method according to claim 24, characterized in that Displaying the content of at least one of the second text blocks and the content of at least one of the fourth text blocks on the second interface includes: The display order of the content of at least one of the second text blocks and the content of at least one of the fourth text blocks is determined according to the eighth information, and the eighth information includes at least one of the following information: spatial information of at least one of the second text blocks in the first interface, hierarchical information of at least one of the second text blocks, spatial information of at least one of the fourth text blocks in the first interface, or hierarchical information of at least one of the fourth text blocks.

26. The method according to any one of claims 21 to 25, characterized in that The method further comprises: A formatting mark is displayed on the second interface, where the formatting mark is used to identify an association relationship between at least one of the second text blocks and / or a hierarchical relationship between at least one of the second text blocks in the first structure.

27. The method according to claim 26, characterized in that The formatting mark includes at least one of the following: brackets, semicolons, dashes, serial numbers, bullets, line breaks, tabs, and vertical bar symbols.

28. The method according to any one of claims 1 to 27, characterized in that The method further comprises: An operation of adjusting the format of the content of the second text block by the user is detected, and the display formats of the text blocks at the same level as the second text block in the second interface are adjusted in batches.

29. An electronic device, characterized in that: include: one or more processors; one or more memories; The one or more memories store one or more computer programs, and the one or more computer programs include instructions, which, when executed by the one or more processors, enable the electronic device to perform the method as claimed in any one of claims 1 to 28.

30. A human-computer interaction device, characterized in that: Comprising means for implementing the method as claimed in any one of claims 1 to 28.

31. A computer-readable storage medium, characterized in that: The storage medium stores a program or an instruction, and when the program or the instruction is executed on a computer, the method according to any one of claims 1 to 28 is executed.

32. A chip, characterized in that: Instructions are stored in the chip, and when the instructions are executed, the method according to any one of claims 1 to 28 is performed.

33. A computer program product, characterized in that The computer program product stores a program or an instruction, and when the program or the instruction is executed, the method according to any one of claims 1 to 28 is executed.

Images