Human-computer interaction method, device and electronic equipment

Abstract

The application provides a human-computer interaction method, device and electronic equipment. In the human-computer interaction method, in response to a text selection operation of a user, the electronic equipment can recognize text content displayed in a first interface, and generate a tree structure of the text content according to a recognition result. Each node in the tree structure is a text block, and a text block of a lower-level child node is located within a text block range of an upper-level parent node. The user selects text in units of text blocks, and the electronic equipment highlights the text block selected by the user in units of text blocks according to the tree structure. When the user pastes text, the electronic equipment displays content of the selected text block in a second interface according to a level of the tree structure. In the technical solution, based on the constructed tree structure, text selection efficiency, user selection intention and reading efficiency after pasting can be considered, and the interaction experience of a text selection process can be improved.

Description

[0001] This application is a divisional application. The original application has the application number 202411331279.1 and the original application date is September 23, 2024. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of electronic equipment technology, and more specifically, to a human-computer interaction method, apparatus, and electronic equipment. Background Technology

[0003] With the continuous development of electronic device technology, electronic devices are becoming increasingly feature-rich. To meet users' needs for sharing and reusing text information, many electronic devices have text extraction functions to facilitate the selection and pasting of text information that cannot be directly edited on the interface. For example, users can select text information by moving the cursor, scribbling, or clicking. However, based on current solutions, users often experience low efficiency, difficulty in fulfilling their selection intentions, or difficulty in reading the pasted text, resulting in a poor user experience. Summary of the Invention

[0004] This application provides a human-computer interaction method, apparatus, and electronic device that can improve the interactive experience of the text extraction process.

[0005] In a first aspect, a human-computer interaction method is provided, comprising: displaying a first interface, the first interface including text content; in response to a first operation by a user, recognizing the text content; generating a tree structure of the text content based on the result of recognizing the text content, the tree structure being used to represent the hierarchical information of the text content, wherein a parent node in the tree structure corresponds to a first text block, and a child node of the parent node corresponds to a second text block, the second text block being located within the area of ​​the first text block; and in response to a second operation by the user, highlighting at least one second text block according to the tree structure.

[0006] In this embodiment, by constructing a tree structure of the text content in the first interface, the text content in the first interface can be hierarchically organized as text blocks, thereby providing a correct reading order and supporting multiple reading orders. Correspondingly, during text extraction, the order in which users select and paste text is more in line with general reading habits, thus satisfying the user's selection intent and improving reading efficiency and the integrity of information dissemination. Furthermore, the method of selecting text based on text blocks is simple to operate and highly efficient. Therefore, the solution provided in this application can balance selection efficiency, user selection intent, and reading efficiency, thereby significantly improving the user's interactive experience.

[0007] In conjunction with the first aspect, in one possible implementation, generating a tree structure of the text content based on the result of recognizing the text content includes: dividing the text content into text blocks according to first information, the first information including at least one of spatial information of the text, semantic information of the text, and visual saliency features in the first interface; determining the hierarchical relationship between the divided text blocks according to second information, the second information including at least one of spatial information of the text, semantic information of the text, and visual saliency features in the first interface; and sorting the divided text blocks according to third information, the third information including spatial information of the text.

[0008] By generating a tree structure based on the first, second, and third pieces of information, precise segmentation and sorting of the original text can be achieved, conforming to the general reading order. In constructing this tree structure, layout analysis based on text location information offers greater real-time performance compared to layout analysis that relies entirely on image information.

[0009] In conjunction with the first aspect, in one possible implementation, the text content is divided into text blocks according to the first information, including: obtaining the first text block; determining at least one candidate segmentation axis according to the fourth information, the candidate segmentation axis passing through the first text block; determining the layout direction of the first text block according to the at least one candidate segmentation axis, wherein the layout direction of the first text block is horizontal or vertical; and dividing the first text block based on the first segmentation axis among the at least one candidate segmentation axis to obtain multiple second text blocks, wherein the extension direction of the first segmentation axis is the same as the layout direction of the first text block.

[0010] Specifically, when the first text block is laid out horizontally, the first dividing axis extends horizontally. When the first text block is laid out vertically, the first dividing axis extends vertically.

[0011] By using the layout direction of the first text block to filter out the actual dividing axis used for segmentation from at least one candidate dividing axis, unsuitable dividing axes can be eliminated, and unreasonable segmentation methods can be avoided.

[0012] In conjunction with the first aspect, in one possible implementation, the fourth information includes at least one of the following: visual saliency features in the first interface, the degree of alignment of the text in the horizontal direction, the degree of alignment of the text in the vertical direction, the mapping relationship between the semantic information of the text and the spatial information of the text, and the layout density of the text.

[0013] There are many ways to obtain candidate dividing axes, which can achieve accurate segmentation of the original text.

[0014] In conjunction with the first aspect, in one possible implementation, the horizontal alignment of the text is determined based on at least one of the following: differences in character size, differences in upper character boundaries, differences in lower character boundaries, and differences in the vertical coordinates of character center points; and / or the vertical alignment of the text is determined based on at least one of the following: differences in character size, differences in left character boundaries, differences in right character boundaries, and differences in the horizontal coordinates of character center points.

[0015] In conjunction with the first aspect, in one possible implementation, at least one candidate segmentation axis includes one or more horizontal candidate segmentation axes and one or more vertical candidate segmentation axes. Determining the layout direction of the first text block based on at least one candidate segmentation axis includes: pre-segmenting the first text block based on one or more horizontal candidate segmentation axes to obtain multiple line text regions; determining the interline layout similarity of the first text block based on fifth information, the fifth information including at least one of the following: alignment difference of line text regions, line width difference of line text regions, text number difference of line text regions, same column text size difference of line text regions, and line spacing difference of line text regions; pre-segmenting the first text block based on one or more vertical candidate segmentation axes to obtain multiple line text regions. A text block is pre-segmented to obtain multiple column text areas; the inter-column layout similarity of the first text block is determined based on the sixth information, which includes at least one of the following: alignment difference of column text areas, column width difference of column text areas, text number difference of column text areas, same-line text size difference of column text areas, and column spacing difference of column text areas; the layout direction of the first text block is determined based on the inter-line layout similarity and the inter-column layout similarity, wherein the layout direction of the first text block is horizontal when the inter-line layout similarity is greater than the inter-column layout similarity; and the layout direction of the first text block is vertical when the inter-column layout similarity is greater than the inter-line layout similarity.

[0016] By calculating the layout similarity to determine the layout direction of text blocks, text blocks can be segmented in a way that conforms to the general reading order.

[0017] In conjunction with the first aspect, in one possible implementation, before segmenting the first text block based on a first segmentation axis among at least one candidate segmentation axes, the method further includes: determining the first segmentation axis from at least one candidate segmentation axis according to the layout orientation of the first text block, wherein, if the layout orientation of the first text block is horizontal, the first segmentation axis is the horizontal candidate segmentation axis; and if the layout orientation of the first text block is vertical, the first segmentation axis is the vertical candidate segmentation axis.

[0018] By using the layout direction of the first text block to filter out the actual dividing axis used for dividing from at least one candidate dividing axis, unsuitable dividing axes can be eliminated, thereby dividing the text block in a way that conforms to the general reading order.

[0019] In conjunction with the first aspect, in one possible implementation, determining at least one candidate segmentation axis based on the fourth information includes: obtaining an adjacency matrix of the first text block, the adjacency matrix including the vertical or horizontal distance between adjacent words in the first text block; determining the position in the first text block corresponding to the maximum value in the adjacency matrix as the candidate segmentation axis; and segmenting the first text block based on the first segmentation axis among the at least one candidate segmentation axis to obtain multiple second text blocks, including: clustering the words on both sides of the first segmentation axis to obtain multiple second text blocks, wherein the vertical or horizontal distance between adjacent words in the second text blocks is less than the maximum value in the adjacency matrix.

[0020] By dividing text blocks based on the adjacency matrix, it can discover clusters of arbitrary shapes and is insensitive to outliers in the dataset.

[0021] In conjunction with the first aspect, in one possible implementation, the method further includes: determining the position in the second text block corresponding to the second maximum value in the adjacency matrix as a second segmentation axis, the second segmentation axis passing through the second text block; segmenting the second text block based on the second segmentation axis to obtain multiple third text blocks, wherein the distance between adjacent words in the third text blocks in the vertical or horizontal direction is less than the second maximum value in the adjacency matrix.

[0022] Based on the adjacency matrix, all clusters can be discovered in a single algorithm execution without needing to pre-specify the number of clusters.

[0023] In conjunction with the first aspect, in one possible implementation, the ratio of the second maximum value to the vertical height of the smallest character within the first text block is greater than or equal to a preset threshold.

[0024] When the ratio of the distance between adjacent words to the size of the smallest character in the first text block is less than a preset threshold, the adjacent words can be considered to be related and should not be divided into different text blocks.

[0025] In conjunction with the first aspect, in one possible implementation, obtaining the first text block includes: obtaining the horizontal alignment of each line of text and / or the vertical alignment of each column of text in the text content; determining multiple consecutive text lines and / or multiple consecutive text columns as the first text block, wherein the horizontal alignment of the text lines is greater than a first threshold, and the vertical alignment of the text columns is greater than a second threshold.

[0026] In conjunction with the first aspect, in one possible implementation, the spatial information of the text includes the position of the text and the size of the text, wherein the position of the text includes the coordinates of the text in the first interface and / or the text line index of the text in the first interface.

[0027] Page layout analysis based on the spatial location of text offers greater real-time performance.

[0028] In conjunction with the first aspect, in one possible implementation, the visual salience feature includes at least one of the following: text color, text font, color block, shadow, and border.

[0029] In conjunction with the first aspect, in one possible implementation, the sorting of the divided text blocks according to the third information includes: when the first text block is divided into multiple second text blocks based on a horizontal dividing axis, sorting the multiple second text blocks from top to bottom according to the spatial information of the multiple second text blocks; or, when the first text block is divided into multiple second text blocks based on a vertical dividing axis, sorting the multiple second text blocks from left to right according to the spatial information of the multiple second text blocks; wherein the sorting priority of the first text block is higher than the sorting priority of the second text block.

[0030] The text blocks obtained through horizontal segmentation are ordered from top to bottom in the same direction as the original text blocks, conforming to the reading order of horizontal layout. Similarly, the text blocks obtained through vertical segmentation are ordered from left to right in the same direction as the original text blocks, conforming to the reading order of vertical layout.

[0031] In conjunction with the first aspect, in one possible implementation, before dividing the text content into text blocks based on the first information, the method further includes: dividing all words in the text content into multiple text lines, each text line horizontally traversing the first interface; sorting the multiple text lines from top to bottom, and obtaining the text line index of each text line, the text line index being used to divide the text content into text blocks and / or sort the text blocks.

[0032] The text line index is the global line number of the text in the first interface, which can be used to determine the relative position between text blocks.

[0033] In conjunction with the first aspect, in one possible implementation, in response to a second user action, highlighting at least one second text block according to the tree structure includes: determining the text block the user intends to select based on the trajectory of the text selected by the user and the tree structure, wherein the trajectory of the text selected by the user passes through the area of ​​at least one second text block, the text block the user intends to select includes at least one second text block; and highlighting at least one second text block.

[0034] Based on a tree structure, the text block selected by the user can be determined according to the trajectory of the selected text. When selecting text, the user can make efficient and logical text selections in a manner consistent with general reading flow, without the problems of interleaved or disordered selections between different structures. When there are multiple possible arrangements of text blocks, the trajectory of the user's text selection can freely express the text selection order that conforms to their intention.

[0035] In conjunction with the first aspect, in one possible implementation, determining the text block that the user intends to select based on the trajectory of the user-selected text and the tree structure includes: detecting that the trajectory of the user-selected text changes from the region of the second text block to the region of the first text block, and marking the hierarchy and sorting information of the second text block selected in the first text block in the tree structure as a template; detecting that the trajectory of the user-selected text changes from the region of the first text block to the region of the sibling node of the parent node, and determining, based on the template, that the text block that the user intends to select includes the text block corresponding to the child node under the sibling node.

[0036] Therefore, inferring the text block selected by the user based on their selection trajectory can better meet the user's selection intent and improve selection efficiency.

[0037] In conjunction with the first aspect, in one possible implementation, the method further includes: highlighting the text blocks corresponding to the child nodes under the sibling node, wherein the visual salience features of the text blocks corresponding to the child nodes under the sibling node are different from the visual salience features of the second text block.

[0038] The visual salience of the selected text block in the second interface is different from that of the second text block in the second interface, which can prompt the user.

[0039] In conjunction with the first aspect, in one possible implementation, the second operation is any of the following: a text selection operation based on the cursor, a text selection operation based on smearing, or a text selection operation based on clicking.

[0040] In conjunction with the first aspect, in one possible implementation, the second operation is a text-based operation that selects text by clicking. Based on the trajectory of the text selected by the user and the tree structure, the operation determines the text block that the user intends to select, including: determining the lowest-level text block corresponding to the user's click position based on the user's click position and the boundary information of the text blocks corresponding to each node in the tree structure; and determining that the lowest-level text block is the text block that the user intends to select.

[0041] In this way, users can select a line of plain text or an item of information with a single click, improving interaction efficiency.

[0042] In conjunction with the first aspect, in one possible implementation, the method further includes: in response to a user's paste operation, displaying the content of at least one second text block on a second interface according to the tree structure.

[0043] When pasting, the electronic device displays at least one second text block on the second interface according to the hierarchy of the tree structure.

[0044] In this embodiment, the tree structure constructed for the text content can provide users with a general reading order. Therefore, when the user pastes the selected text block content onto the second interface, the selected text block will also display relevant information in an order that conforms to reading habits.

[0045] In conjunction with the first aspect, in one possible implementation, in response to a user's paste operation, displaying the content of at least one second text block on a second interface according to the tree structure includes: displaying the content of at least one second text block in the same row on the second interface, wherein at least one second text block satisfies a first preset condition; the first preset condition includes at least one of the following: at least one second text block has the same parent node; at least one second text block has the same association relationship, which is a subordinate relationship, a parallel relationship, or a start-end relationship.

[0046] Child nodes with the same parent node are usually close in location and will appear together in the user's small reading area. Grouping text blocks with the same parent node together in one line conforms to the user's reading order. Text blocks belonging to the same relationship are close in reading order; displaying texts with the same relationship together conforms to the reading order and improves reading efficiency.

[0047] In conjunction with the first aspect, in one possible implementation, displaying at least one second text block in the same row in the second interface includes: determining the display order of the content of at least one second text block according to seventh information, the seventh information including at least one of the following: spatial information of at least one second text block in the first interface, semantic information of at least one second text block, or at least one of the visual salience features related to at least one second text block in the first interface.

[0048] When different text blocks are displayed on the same line, their order can be rearranged, which can improve reading efficiency.

[0049] In conjunction with the first aspect, in one possible implementation, in response to a user's paste operation, displaying the content of at least one second text block on a second interface according to the tree structure includes: displaying the content of at least one second text block and the content of at least one fourth text block on the second interface, wherein the content of the second text block and the content of the fourth text block are located on different lines of the second interface, and the second text block and the fourth text block satisfy a second preset condition; the second preset condition includes at least one of the following: the second text block and the fourth text block have different parent nodes; the second text block and the fourth text block have different association relationships, including subordinate relationship, parallel relationship, and start-end relationship.

[0050] Displaying child nodes with different parent nodes or with different relationships on separate lines can highlight the structural hierarchy between text nodes and help improve the efficiency of information dissemination.

[0051] In conjunction with the first aspect, in one possible implementation, displaying the content of at least one second text block and the content of at least one fourth text block on the second interface includes: determining the display order of the content of at least one second text block and the content of at least one fourth text block according to eighth information, the eighth information including at least one of the following: spatial information of at least one second text block in the first interface, hierarchical information of at least one second text block, spatial information of at least one fourth text block in the first interface, or hierarchical information of at least one fourth text block.

[0052] This application reorganizes the text blocks according to their position and hierarchy, which helps to restore the structural hierarchy and relationships contained in the original layout and improves reading efficiency.

[0053] In conjunction with the first aspect, in one possible implementation, the method further includes: displaying formatting tags on the second interface, the formatting tags being used to identify the association between at least one second text block and / or the hierarchical relationship of at least one second text block in the tree structure.

[0054] Adding special symbols helps to restore the structural hierarchy and relationships contained in the original layout, improving the efficiency and completeness of information dissemination.

[0055] In conjunction with the first aspect, in one possible implementation, the formatting markup includes at least one of the following: parentheses, semicolons, dashes, serial numbers, bullets, line breaks, tabs, and vertical lines.

[0056] In conjunction with the first aspect, in one possible implementation, the method further includes: detecting a user's operation to adjust the format of the content of the second text block, and batch adjusting the display format of text blocks at the same level as the second text block in the second interface.

[0057] Electronic devices can provide users with the function of batch adjusting the display format of text blocks at the same level, which can improve the efficiency of format adjustment.

[0058] In conjunction with the first aspect, in one possible implementation, the granularity of the second text block is smaller than that of the first text block.

[0059] In conjunction with the first aspect, in one possible implementation, among the multiple child nodes of the parent node, adjacent child nodes are bidirectionally interconnected.

[0060] In conjunction with the first aspect, in one possible implementation, the leaf nodes in this tree structure correspond to words or lines of text.

[0061] In conjunction with the first aspect, in one possible implementation, the first interface displays at least one of the following: an image, a webpage, or a document.

[0062] In conjunction with the first aspect, in one possible implementation, the text content is uneditable.

[0063] In a second aspect, an apparatus is provided, which is included in an electronic device and has the functionality to perform the behaviors involved in the first aspect and any possible implementation thereof.

[0064] This function can be implemented through hardware or by executing corresponding software within the hardware. The hardware or software includes one or more modules or units corresponding to the above function. Examples include display modules or units, recognition modules or units, generation modules or units, partitioning modules or units, sorting modules or units, acquisition modules or units, detection modules or units, processing modules or units, etc.

[0065] Thirdly, an electronic device is provided, comprising: one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0066] Fourthly, a computer-readable storage medium is provided, including computer instructions that, when executed on an electronic device, cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0067] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the methods described in the first aspect and any possible implementation thereof.

[0068] In a sixth aspect, a chip is provided, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface and executing the methods in the first aspect and any possible implementation thereof.

[0069] Alternatively, as one implementation, the chip may also include a memory storing instructions, and a processor for executing the instructions stored in the memory. When the instructions are executed, the processor is used to perform the methods in the first aspect and any possible implementation thereof.

[0070] The aforementioned chip can be a field-programmable gate array or an application-specific integrated circuit.

[0071] The beneficial effects of the apparatus described in the second to sixth aspects can be referred to the beneficial effects of the method described in the first aspect, and will not be repeated here. Attached Figure Description

[0072] Figure 1 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0073] Figure 2 This is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application.

[0074] Figure 3-5 This is a schematic diagram of the existing text extraction interface.

[0075] Figure 6 This is a schematic flowchart of a human-computer interaction method provided in an embodiment of this application.

[0076] Figure 7 This is a partial schematic diagram of a tree structure provided in an embodiment of this application.

[0077] Figure 8 This is a partial schematic diagram of a tree structure provided in an embodiment of this application.

[0078] Figure 9 This is a schematic flowchart illustrating the generation of a tree structure in the human-computer interaction method provided in this application embodiment.

[0079] Figure 10-13 This is a schematic diagram of determining the position of the segmentation axis in the human-computer interaction method provided in this application.

[0080] Figure 14 This is a schematic diagram of word features provided in the embodiments of this application.

[0081] Figure 15 This is a schematic diagram of the text block layout direction provided in the embodiments of this application.

[0082] Figure 16-18 This is a schematic diagram illustrating the determination of layout direction in the human-computer interaction method provided in this application embodiment.

[0083] Figure 19-21 This is a schematic diagram of text selection in the human-computer interaction method provided in the embodiments of this application.

[0084] Figure 22 This is a partial schematic diagram of a tree structure provided in an embodiment of this application.

[0085] Figure 23-24 This is a schematic diagram of text selection in the human-computer interaction method provided in the embodiments of this application.

[0086] Figure 25-29 This is a schematic diagram of pasting text in the human-computer interaction method provided in the embodiments of this application.

[0087] Figure 30 This is a schematic structural block diagram of a device provided in an embodiment of this application.

[0088] Figure 31 This is a schematic structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0089] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0090] It should be noted that, in the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in this article is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0091] In the embodiments of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more, and "at least one" and "one or more" refer to one, two, or more than two. The singular expressions "a," "an," "the," "the," "this," and "this" are intended to also include expressions such as "one or more," unless the context explicitly indicates otherwise.

[0092] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0093] To facilitate understanding, some technical terms involved in the embodiments of this application will be explained and described below.

[0094] Tree structure: A hierarchical structure defined by branching relationships, which is a finite set of n nodes. The basic unit of a tree structure is a node, and the links between nodes are called branches. Nodes and branches form a tree structure.

[0095] Root node: The top-level node, which is the beginning of the tree structure.

[0096] Leaf nodes (or terminal nodes): These are the outermost nodes and represent the end of the tree structure. Leaf nodes only have a predecessor node and no successor node.

[0097] Parent node: The node above a given node (i.e., the predecessor node) is called the parent node of that node.

[0098] Child node: The next level node (i.e., the successor node) of a given node is called the child node of that node.

[0099] Sibling nodes: Nodes that share the same parent node are sibling nodes.

[0100] Ancestor nodes: All nodes on the branches from the root node to a given node are called the ancestor nodes of that node.

[0101] Descendant node: Any node in the subtree rooted at a given node is called a descendant node of that node.

[0102] Cousin nodes: Nodes whose parent node is on the same level are cousin nodes.

[0103] Subtree: A tree rooted at the child nodes of a given node is called the subtree of that node.

[0104] Node hierarchy: Starting from the root node, the root node is the first level, the children of the root are the second level, and if a node is at level L, then the root of its subtree is at level L+1.

[0105] Parent-child relationship: In a hierarchical structure, the relationship between a node and its direct child nodes is defined as a parent-child relationship. There is an inclusion relationship between a parent node and its child nodes; the parent node contains information about its child nodes.

[0106] Sibling relationship: Sibling nodes are all child nodes that share the same parent node. These nodes have an equivalence relationship, meaning they share the same parent node but are not contained within each other.

[0107] Superior-subordinate relationship: In a hierarchical structure, a superior-subordinate relationship refers to the relationship between a node and its indirect parent node. This relationship is often used to represent the connection between entities with multi-level containment relationships.

[0108] Visual saliency features refer to regions or objects in a visual scene that attract human attention. In real-world scenes, people automatically identify regions of interest and process them, ignoring uninteresting regions. In computer vision, visual saliency is defined as using computer technology to simulate the human visual system's perception of images, extracting salient regions (i.e., regions of interest) to quickly search for and locate targets of interest in natural scenes.

[0109] Density-based spatial clustering of applications with noise (DBSCAN) defines a cluster as the largest set of density-connected points. It can divide regions with sufficiently high density into clusters and can discover clusters of arbitrary shapes in noisy spatial databases.

[0110] Optical character recognition (OCR) is a technology that scans and analyzes text in images or documents, converting it into editable and searchable data.

[0111] Page layout: refers to the design, typesetting, and organization of a page, including the position and distribution of elements such as text and images.

[0112] The method provided in this application can be applied to electronic devices with display functions, such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), smart home devices (e.g., smart screens), etc. This application does not impose any restrictions on the specific type of electronic device.

[0113] For example, Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application is shown.

[0114] like Figure 1 As shown, the electronic device 100 may include: a processor 110, a memory 120, a universal serial bus (USB) interface 130, a power supply 140, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a microphone 170C, a sensor module 180, a camera 191, a display screen 192, etc.

[0115] Processor 110 may include one or more processing units. For example, processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. The different processing units may be independent devices or integrated into one or more processors.

[0116] The controller can serve as the nerve center and command center of an electronic device. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0117] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from this memory, avoiding repeated accesses, reducing the waiting time of the processor 110, and thus improving the efficiency of the system.

[0118] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0119] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0120] For example, processor 110 and touch sensor 180B can communicate via an I2C bus interface to enable touch functionality in the electronic device. Processor 110 and camera 191 can communicate via a CSI interface to enable image capture functionality in the electronic device. Processor 110 and display screen 192 can communicate via a DSI interface to enable display functionality in the electronic device.

[0121] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0122] Power supply 140 is used to supply power to processor 110, memory 120, display 192, camera 191, mobile communication module 150 and wireless communication module 160, etc.

[0123] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0124] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0125] The mobile communication module 150 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G in electronic devices. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0126] The wireless communication module 160 can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0127] Electronic device 100 implements display functions through GPU, display screen 192, and application processor.

[0128] A GPU is a microprocessor for image processing, connected to the display screen 192 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0129] The display screen 192 is used to display images, videos, etc. The display screen 192 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 192, where N is a positive integer greater than 1.

[0130] Electronic device 100 can perform shooting functions through ISP, camera 191, video codec, GPU, display 192 and application processor.

[0131] The ISP is used to process data fed back by camera 191. Camera 191 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. In some embodiments, the electronic device may include one or N cameras 191, where N is a positive integer greater than 1.

[0132] Video codecs are used to compress or decompress digital video. Electronic devices can support one or more video codecs. This allows the electronic device to play or record video in various encoded formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0133] The memory 120 is used to store data and / or instructions.

[0134] Memory 120 may include internal memory. Internal memory stores computer-executable program code, including instructions. Processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory. Internal memory may include a program storage area and a data storage area. The program storage area may store the operating system; it may also store one or more applications (such as a gallery, contacts, etc.). The data storage area may store data created during the use of the electronic device (such as images, contacts, etc.). Furthermore, internal memory may include high-speed random access memory and non-volatile memory, such as one or more disk storage devices, flash memory devices, universal flash storage (UFS), etc. In some embodiments, processor 110 may cause the electronic device to perform the methods provided in the embodiments of this application by running instructions stored in the internal memory and / or instructions stored in memory disposed within processor 110.

[0135] The memory 120 may also include external memory, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory can communicate with the processor 110 via an external memory interface to perform data storage functions. For example, music, video, and other files can be stored in the external memory.

[0136] Electronic devices can implement audio functions, such as audio playback and recording, through audio module 170, speaker 170A, microphone 170C, and application processor.

[0137] The sensor module 180 may include a pressure sensor 180A, a touch sensor 180B, a gyroscope sensor, an accelerometer sensor, a distance sensor, and other sensors.

[0138] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 192. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. The electronic device determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 192, the electronic device detects the touch operation intensity based on pressure sensor 180A. The electronic device can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example: when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0139] Touch sensor 180B, also known as a "touch panel," can be located on display screen 192. The touch sensor 180B and display screen 192 together form a touchscreen, also known as a "touchscreen." Touch sensor 180B detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 192. In other embodiments, touch sensor 180B may also be located on the surface of the electronic device, in a different position than display screen 192.

[0140] Gyroscope sensors are used to determine the motion and attitude of electronic devices. For example, gyroscope sensors can be used for image stabilization, navigation, and motion-sensing games. When the electronic device is stationary, the gyroscope sensor can detect the magnitude and direction of gravity.

[0141] Accelerometers can detect the magnitude of acceleration in various directions (typically three axes) of electronic devices.

[0142] A distance sensor is used to measure distance. Electronic devices can measure distance using infrared or laser. In some embodiments, during a shooting scene, the electronic device can utilize a distance sensor to measure distance for rapid focusing.

[0143] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0144] The above provides schematic diagrams of possible hardware structures for electronic devices. The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application's embodiment uses a layered architecture. Taking this system as an example, the software structure of the electronic device is illustrated. However, it is understood that the software system of the electronic device in this embodiment can also be other systems, such as HarmonyOS. system, Systems, etc., will not be described in detail here.

[0145] Figure 2 A software structure block diagram of an electronic device provided in an embodiment of this application is shown. Figure 2 As shown, a layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, [the following is omitted as the text is incomplete and likely refers to a specific implementation or feature]. The system is divided into four layers, from top to bottom: application layer, application framework layer, system runtime library layer, and kernel layer.

[0146] The application layer can include a series of application packages. For example... Figure 2 As shown, the application package can include applications such as camera, gallery, calendar, call, map, weather, WLAN, Bluetooth, music, video, SMS, and voice assistant. The applications primarily concern the user interface (UI), and are typically written using the Java language to call the application framework layer's interfaces.

[0147] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 2 As shown, the application framework layer may include a window manager, content provider, phone manager, resource manager, notification manager, view system, etc.

[0148] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0149] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0150] A phone manager is used to provide communication functions for electronic devices. For example, it manages call status (including connection and disconnection).

[0151] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0152] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating the device, and flashing indicator lights.

[0153] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0154] The system runtime library layer is a collection of libraries located below the application framework layer. It can be divided into two parts: system libraries and application runtime (e.g., Android runtime).

[0155] The application runtime consists of the core libraries and the virtual machine. The application runtime is responsible for the scheduling and management of the software system. The core libraries comprise two parts: one part contains the functionalities that the Java language needs to call, and the other part contains the core libraries of the software system.

[0156] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0157] System libraries support the application framework and can include multiple functional modules, such as: surface manager, media libraries, 2D graphics engine (e.g., SGL), 3D graphics processing library (e.g., OpenGL ES), image processing library, etc.

[0158] The Interface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0159] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0160] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0161] A 2D graphics engine is a graphics engine for 2D drawing.

[0162] The kernel layer is the layer between hardware and software, providing essential operating system functions such as file management, memory management, process management, and network protocol stacks. The kernel layer includes at least display drivers, camera drivers, audio drivers, sensor drivers, and Bluetooth drivers.

[0163] For ease of understanding, the following embodiments of this application will be described using the following methods: Figure 1 and Figure 2 Taking the electronic device with the structure shown as an example, and in conjunction with the accompanying drawings and application scenarios, the human-computer interaction method provided in this application embodiment will be specifically described.

[0164] With the continuous development of electronic device technology, electronic devices are becoming increasingly feature-rich. To meet users' needs for sharing and reusing text information, many electronic devices have text extraction functions, allowing users to select and paste text information that cannot be directly edited on the interface. For example, users can select text information by moving the cursor, scribbling, or clicking. However, based on current solutions, the user experience when using text extraction functions is poor.

[0165] For example, in text selection based on the cursor, issues such as multiple selections, omissions, and incorrect selections often occur. Figure 3 A schematic diagram of the interface for cursor-based text selection in the existing solution is shown.

[0166] refer to Figure 3As shown in (a), the content in the dashed box is the text that the user intends to select (e.g., multiple trip information). However, when dragging the cursor to select text, only part of the text is selected, resulting in the loss of local information and incomplete selection.

[0167] refer to Figure 3 As shown in (b), the content in the dashed box is the text that the user intends to select. However, when dragging the cursor, the cursor jumps around randomly. For example, the position of the adjusted (i.e., dragged) cursor jumps, or the position of the unadjusted (i.e., not dragged) cursor jumps, resulting in the situation shown in the figure where the start cursor and end cursor are flipped (i.e., the order is reversed), making it difficult for the user to control and thus causing the wrong content to be selected.

[0168] refer to Figure 3 As shown in (c) and (d), the content in the dashed box is the text that the user intends to select. However, when dragging the cursor to select the text in the dashed box, other text outside the dashed box is also selected, resulting in multiple selection.

[0169] The cursor-based text selection method relies on the system's own text sorting. The aforementioned omissions and under-selections mainly occur because errors in the system's page layout analysis lead to disordered text sorting, preventing users from selecting text in the normal reading order. The aforementioned over-selection also occurs, in part, due to errors in the system's page layout analysis causing disordered text sorting, such as... Figure 3 As shown in (c); another reason is that the system only supports one reading order for the page layout analysis results, so users can only select text in one order. Figure 3 Taking (d) as an example, the itinerary section on the page actually has two potential reading orders: one is a horizontal reading order, which focuses on both outbound and return itineraries; the other is a vertical reading order, which focuses only on either the outbound or return itinerary. If the system only supports a horizontal reading order, when a user intends to select the content in dashed box 1, they will select multiple return information; if the system only supports a vertical reading order, when a user intends to select the content in dashed box 2, they will select multiple outbound information.

[0170] In conclusion, the method of selecting text based on the cursor cannot well meet the user's selection intention and has a poor interactive experience.

[0171] For example, in text selection methods based on smearing or clicking, users need to select or deselect words one by one, which is inefficient. Figure 4 The diagram shows an interface for text selection based on smearing or clicking in an existing solution.

[0172] refer to Figure 4As shown in (a), one approach involves the system displaying the text content of the page on individual cards, allowing users to select words one by one in any order by clicking or scribbling. In this method, the cards cover part of the page, and the text layout on the cards differs from the original page layout. Therefore, when selecting and adjusting text, users need to compare the text with the original image and precisely select or deselect each item, significantly impacting operational efficiency and increasing the cognitive load of the interaction.

[0173] refer to Figure 4 As shown in (b), another approach involves the system highlighting text boxes on the original page, allowing users to select words one by one in any order by clicking or scribbling. In this method, users need to precisely select or deselect, and the small-scale clicking or scribbling increases the complexity of the operation, requiring precise user input and resulting in low efficiency.

[0174] In summary, selecting text based on smearing or clicking is cumbersome, inefficient, and provides a poor user experience.

[0175] In addition, the process of pasting text to other locations after selecting it also suffers from a poor user experience. Figure 5 A schematic diagram of the interface for selecting and pasting text in an existing solution is shown.

[0176] refer to Figure 5 As shown in (a), taking the extraction of itinerary information as an example, the related text in the original layout, such as start and end times and location information, is broken up after pasting, with seat class information interspersed among them.

[0177] refer to Figure 5 As shown in (b) in the example of extracting weather information, texts that are on the same line in the original layout are cross-sorted after pasting, resulting in a chaotic correspondence.

[0178] The reason for this is that the display order of the text during pasting matches the system's sorting of the recognized text. If the system's analysis of the layout is incorrect, causing the text to be sorted out and not conforming to the general reading order, the pasted text will also fail to reflect the original layout's reading order, reducing reading efficiency and the integrity of information dissemination, resulting in a poor interactive experience.

[0179] In summary, existing text extraction solutions generally suffer from poor user experience. Therefore, this application provides a human-computer interaction method that improves the user experience of extracting text from webpage content.

[0180] Figure 6 A schematic flowchart of a human-computer interaction method provided in an embodiment of this application is shown. Figure 6The method 200 shown is applied to an electronic device that may have, for example... Figure 1 The hardware structure shown and / or as shown Figure 2 The software architecture is shown. Method 200 includes steps S210 to S240.

[0181] S210, Display a first interface, which includes text content.

[0182] In this embodiment, the text content on the first interface cannot be directly edited. For example, the first interface displays images, web pages, or documents (such as PDF documents) containing text content.

[0183] S220, in response to the user's first action, identifies the text content.

[0184] For example, the first operation could be clicking a control used for text recognition, or long-pressing the area containing the text content. This first operation can trigger the character selection process.

[0185] In this step, the recognition result of the text content includes: words in the text content, and spatial information of the words in the text content (e.g., position and size). The recognition result of the text content can be selected and pasted. The text content processing involved in the following steps of this application is actually achieved by processing the recognition result of the text content.

[0186] This can be understood as step S220 being used to convert non-editable text content into editable or searchable text content (such as selectable and pasteable).

[0187] In some embodiments, the electronic device recognizes the text content on the first interface based on Optical Character Recognition (OCR) technology. Accordingly, the recognition result of the text content is the OCR recognition result.

[0188] In some embodiments, in step S220, in response to the user's first operation, the electronic device can also acquire visually salient features in the first interface, such as text color, font, border, color block, shadow, etc.

[0189] S230, based on the result of recognizing the text content, a tree structure of the text content is generated. The tree structure is used to represent the hierarchical information of the text content. The parent node in the tree structure corresponds to the first text block, and the child node of the parent node corresponds to the second text block. The second text block is located within the area of ​​the first text block.

[0190] In this application, the text content is divided into multiple text blocks, each text block corresponding to a node in a tree structure. The text block corresponding to a child node is located within the area of ​​the text block corresponding to the parent node, and the content of the text block of the parent node includes the content of the text blocks of its child nodes.

[0191] In this application, a parent node in the tree structure includes multiple child nodes. Correspondingly, the first text block corresponding to the parent node includes multiple second text blocks.

[0192] S240, in response to a second user action, highlights at least one second text block according to a tree structure.

[0193] The second operation is the selection of text. For example, the second operation can be selecting text based on the cursor, selecting text based on swiping, or selecting text based on clicking. More specifically, the second operation can be a long press, a swipe, or a click.

[0194] In some embodiments, the way to highlight at least one second text block may include at least one of the following: highlighting, adding a border line, or adding an underline.

[0195] In this embodiment, by constructing a tree structure of the text content in the first interface, the text content in the first interface can be hierarchically organized as text blocks, thereby providing a correct reading order and supporting multiple reading orders. Correspondingly, during text extraction, the order in which users select and paste text is more in line with general reading habits, thus satisfying the user's selection intent and improving reading efficiency and the integrity of information dissemination. Furthermore, the method of selecting text based on text blocks is simple to operate and highly efficient. Therefore, the solution provided in this application can balance selection efficiency, user selection intent, and reading efficiency, thereby significantly improving the user's interactive experience.

[0196] For ease of understanding, Figure 7 and Figure 8 Partial schematic diagrams of the tree structure are shown with specific examples. It can be understood that... Figure 7 and Figure 8 For illustrative purposes only, each of these figures represents a subtree within a tree structure.

[0197] refer to Figure 7Text block 'a' can be divided into text blocks 'b' and 'c'. In the tree structure, text block 'a' is the parent node of text blocks 'b' and 'c', and text blocks 'b' and 'c' are sibling nodes. Text block 'b' can be further divided into text blocks 'd' and 'e', ​​and text block 'c' can be further divided into text blocks 'g1', 'g2', and 'g3'. In the tree structure, text block 'b' is the parent node of text blocks 'd' and 'e', ​​and text blocks 'd' and 'e' are sibling nodes. Text block 'c' is the parent node of text blocks 'g1', 'g2', and 'g3', and text blocks 'g1', 'g2', and 'g3' are sibling nodes. The child nodes of text block 'b' and text block 'c' are cousin nodes. Text block 'd' can be further divided into text blocks 'f1', 'f2', 'f3', 'f4', 'f5', and 'f6'. In the tree structure, text block d is the parent node of text blocks f1, f2, f3, f4, f5, and f6, and text blocks f1, f2, f3, f4, f5, and f6 are sibling nodes. Text blocks e, f1, f2, f3, f4, f5, f6, g1, g2, and g3 each have only predecessor nodes and no successor nodes, and are therefore leaf nodes in the tree structure.

[0198] refer to Figure 8 Text block 'a' can be divided into text blocks 'b' and 'c'. In the tree structure, text block 'a' is the parent node of text blocks 'b' and 'c', and text blocks 'b' and 'c' are sibling nodes. Text block 'c' can be further divided into text blocks 'd1', 'd2', 'd3', 'd4', and 'd5', and text blocks 'd1', 'd2', 'd3', 'd4', and 'd5' are sibling nodes. Text blocks 'd1', 'd2', 'd3', 'd4', and 'd5' can each be further divided into smaller text blocks. For example, text block 'd1' can be divided into text blocks 'e1', 'e2', 'e3', 'e4', 'e5', and 'e6'; text block 'd2' can be divided into text blocks 'f1', 'f2', 'f3', 'f4', 'f5', and 'f6'; and text block 'd5' can be divided into text blocks 'g1', 'g2', 'g3', 'g4', 'g5', and 'g6'. Text blocks 'd1', 'd2', 'd3', 'd4', and 'd5' are the parent nodes of their child nodes. The child nodes of text blocks d1, d2, d3, d4, and d5 are cousin nodes.

[0199] Figure 7 and Figure 8 In the example, each parent node is a specific instance of the first text block, and the child nodes under the parent node are specific instances of the second text block.

[0200] As shown in the diagram, the text block of a child node is located within the text block area of ​​its parent node, and the content of the parent node's text block includes the content of its child node's text block. Thus, based on this tree-like hierarchical structure, users can select text block by block, choosing individual lower-level or higher-level text blocks, thereby supporting a more flexible text selection order. Furthermore, the hierarchical relationship between text blocks aligns better with typical reading order.

[0201] The text blocks involved in the embodiments of this application are texts with a hierarchical structure, which can also be called text structures.

[0202] The following section, in conjunction with the accompanying drawings, provides a more detailed explanation of the process of generating the tree structure in step S230.

[0203] refer to Figure 9 In some embodiments, step S230 may specifically include steps S231 to S233.

[0204] S221, Based on the first information, divide the text content into text blocks.

[0205] The first information may include at least one of the following: spatial information of the text, semantic information of the text, or visual salient features in the first interface.

[0206] In some embodiments, the spatial information of the text may include the position and size of the text. The position of the text may include the coordinates of the text within the first interface and / or the line index of the text within the first interface.

[0207] For example, the coordinates of the text in the first interface can be corner coordinates, boundary coordinates, or center point coordinates.

[0208] For example, the text line index in the first interface is obtained by dividing all the words in the text content into lines and sorting them from top to bottom. The text line index can be used to determine the relative position of two text blocks, such as whether the two text blocks are located on the same text line in the first interface.

[0209] For example, the size of the text may include the text's vertical dimension and horizontal dimension.

[0210] It should be noted that the “text” mentioned in this application can be characters, words, sentences, paragraphs, chapters, or text lines.

[0211] In some embodiments, the semantic information of the text may include at least one of the following: content features (i.e., the specific information contained in the text, such as topics, entities, etc.), semantic features (i.e., the meaning and contextual relationships of the text, such as word meaning, sentence meaning, etc.), structural features (i.e., the layout and organization of the text, such as paragraph division, headings, etc.), functional features (i.e., the purpose and function of the text, such as informational, entertaining, etc.), and temporal features (i.e., the time state in which the events described in the text occur, such as past, present, future, etc.).

[0212] In some embodiments, the visually salient features in the first interface may include at least one of the following: text color, text font, color block, shadow, border, etc.

[0213] In this embodiment, when dividing text blocks, the entire page is taken as the largest text block. This largest text block is first divided into multiple smaller text blocks, and then each smaller text block is further divided into even smaller text blocks until the smallest text block cannot be divided, thus completing the text block division. Therefore, the divided text blocks are in a parallel or nested relationship.

[0214] In this embodiment, the first information is used to determine the segmentation axis when dividing a large text block. The following example illustrates this using the first text block of the parent node and the second text block of its child node.

[0215] As an example, the first information includes visually salient features in the first interface, such as background color differences, shadows, borders, color blocks, etc., and the segmentation axis can be determined based on these visually salient features.

[0216] For example, refer to Figure 10 In (a), if there is a straight line passing through the first text block B1 within the region of the first text block B1, then the first text block B1 can be divided into multiple second text blocks B2 by using the straight line as the dividing axis.

[0217] For example, refer to Figure 10 In (b) or (c), if there are borders within the area of ​​the first text block B1, the first text block B1 can be divided into multiple second text blocks B2 by using the position between adjacent borders as the dividing axis.

[0218] For example, refer to Figure 10 In (d), if there are color blocks within the area of ​​the first text block B1, the first text block B1 can be divided with the boundary of the color blocks as the dividing axis to obtain multiple second text blocks B2.

[0219] As another example, the first information includes the semantic information of the text, which can be used to determine the semantic relationship between the texts, and the segmentation axis can be determined based on the semantic relationship between the texts.

[0220] For example, refer to Figure 10 In (c), the first text block B1 includes outbound and return information. "Outbound" and "return" are semantically related, implying that they can be separated. Therefore, the first text block B1 can be divided into multiple second text blocks B2 by using the position between the text related to outbound information and the text related to return information as the dividing axis.

[0221] For example, refer to Figure 10In (d) of the first text block B1, the text on the left is all time-related, which is significantly different from the text entity type on the right, semantically implying that it is separable. Therefore, the first text block B1 can be divided into multiple second text blocks B2 by using the position between the time-related text on the left and the non-time-related text on the right as the dividing axis.

[0222] For example, refer to Figure 11 In (a) of the first text block B1, the text on the right side all belong to price entities, which are significantly different in type from the text entities on the left side, semantically implying that they are separable. Therefore, the first text block B1 can be divided into multiple second text blocks B2 by using the position between the price entity text on the right and the price entity text on the left as the dividing axis.

[0223] For example, refer to Figure 11 In (b) of the first text block B1, the text entities in the two text areas at similar positions have the same type and present a semantic-spatial pattern with a high degree of similarity. Therefore, the first text block B1 can be divided into multiple second text blocks B2 by using the position between the two similar pages as the dividing axis.

[0224] As another example, the first piece of information includes spatial information about the text, which can be used to determine the text's horizontal alignment (or lateral alignment), vertical alignment (or vertical alignment), or text density (such as horizontal or vertical text density). The horizontal alignment, vertical alignment, or text density can be used to determine the dividing axis. The text density can be determined based on the spacing between adjacent words in the text.

[0225] For example, refer to Figure 12 In (a) and (b) of the above, if there are text areas with high vertical alignment (e.g., left-aligned, right-aligned, or center-aligned), then the boundary of these text areas with high vertical alignment can be used as the dividing axis to divide the first text block B1 into multiple second text blocks B2. For example... Figure 12 As shown in (a), if the text area to the left of the dividing axis is right-aligned vertically, then the right boundary of that text area can be used as the dividing axis. Figure 12 As shown in (b), the text area to the right of the dividing axis is vertically left-aligned, so the left boundary of the text area can be used as the dividing axis.

[0226] For example, refer to Figure 12 In (c), if there is a large gap formed by two text areas in the first text block B1, then the position of this gap can be used as the dividing axis to divide the first text block B1 into multiple second text blocks B2.

[0227] For example, refer to Figure 13 In step (a), the adjacency matrix of the first text block B1 can be obtained first. This adjacency matrix includes the vertical distance between adjacent words in the first text block (or the horizontal distance between adjacent words in the first text block). In some embodiments, the elements in the adjacency matrix are also called spacing values, which are text spacings, such as line spacing (or column spacing). In this embodiment, the smaller the spacing value, the higher the density, and the larger the spacing value, the lower the density. By calculating the changing trend of the vertical (or horizontal) distance between adjacent words in the adjacency matrix, values ​​with significant abrupt increases in vertical (or horizontal) distance can be found. These values ​​can be marked as density drop points (i.e., the positions where text spacing increases). There may be multiple density drop points in the first text block B1, such as 16, 38, and 108 shown in the figure. Therefore, when segmenting the first text block B1, a round of DBSCAN clustering is first performed on all words in the first text block B1 using the density drop point with the largest spacing value (such as 108). DBSCAN clustering groups words with adjacent densities greater than the density corresponding to the density drop point together, and words with adjacent densities less than the density corresponding to the density drop point into different clusters. In other words, words whose vertical (or horizontal) distance from adjacent words is less than the spacing value corresponding to the density drop point are clustered in one text block, and words whose vertical (or horizontal) distance from adjacent words is greater than the spacing value corresponding to the density drop point are clustered in different text blocks.

[0228] like Figure 13 As shown in (b), based on DBSCAN clustering, the first text block B1 can be divided into multiple second text blocks B2 according to the maximum value of the density drop point (e.g., 108), where the position of the density drop point is the position of the segmentation axis.

[0229] The DBSCAN clustering algorithm is an AI algorithm model that clusters points based on the distances between them. Two important parameters determine the clustering results: the distance matrix and the maximum neighbor distance (eps). In this example, the distance matrix input to the algorithm model is the adjacency matrix between words mentioned above, and eps is the value of the points where density drops sharply.

[0230] As mentioned above, the adjacency matrix may have multiple density drop points. After dividing the first text block B1 into multiple second text blocks B2 through one round of DBSCAN clustering, DBSCAN clustering can be continued on the second text blocks B2 based on the second maximum value of the density drop points (e.g., 38). For example... Figure 13 As shown in (c), based on DBSCAN clustering, each second text block B2 can be divided into multiple third text blocks B3 according to the second maximum value of the density drop point (e.g., 38), where the position of the density drop point is the position of the segmentation axis.

[0231] In other words, the adjacency matrix of the first text block B1 can determine the segmentation axes of multiple levels of text blocks for multi-round segmentation. Specifically, the adjacency matrix of the first text block B1 may have 0 to multiple density drop points. For all density drop points, DBSCAN clustering can be performed on all words within the text block in descending order of spacing value, based on each density drop point. In each round of DBSCAN clustering, the algorithm can cluster words with adjacent densities greater than the density corresponding to the density drop point together to form new text blocks. Therefore, after each round of clustering, the original text block will be divided into several smaller text blocks. With multiple rounds of DBSCAN clustering based on smaller density drop points, the text block obtained in the previous iteration is divided into smaller text blocks until all density drop points have been clustered.

[0232] In some embodiments, if the ratio of the value of a density drop point in the adjacency matrix to the size (e.g., vertical dimension) of the smallest character in the first text block B1 is less than a preset threshold, the text block will no longer be segmented based on the value of that density drop point. This is because the smaller the value of the density drop point, the higher the word density and the smaller the distance between adjacent words. When the ratio of the distance between adjacent words to the size of the smallest character in the first text block B1 is less than the preset threshold, the adjacent words can be considered to have a relationship and should not be segmented into different text blocks.

[0233] In some embodiments, if the distances between adjacent words in the first text block B1 are all equal, then there are 0 density drop points in the first text block B1, but a round of DBSCAN clustering can be performed on the first text block B1 based on the maximum value (all of which are maximum values) in the adjacent matrix.

[0234] In this embodiment of the application, in order to divide a larger text block into smaller text blocks, the dividing axis determined according to the first information must penetrate through the text block being divided. For example, the dividing axis used to divide the first text block B1 into multiple second text blocks B2 must penetrate through the first text block B1.

[0235] In some embodiments, the horizontal alignment of the text involved in this application can be determined based on at least one of the following: differences in character size, differences in upper character boundaries, differences in lower character boundaries, and differences in the vertical coordinates of the character center points.

[0236] In some embodiments, the vertical alignment of the text can be determined based on at least one of the following: differences in character size, differences in left margins between characters, differences in right margins between characters, and differences in the horizontal coordinates of the center points between characters.

[0237] For ease of understanding, please refer to Figure 14As shown in (a), a character has a border that encloses it, consisting of a top edge, a bottom edge, a left edge, and a right edge. The center point of this border is the center point of the character. (See reference) Figure 14 As shown in (b), the horizontal alignment of text describes the vertical differences between individual characters in a line of text. For example, the horizontal alignment can be determined by the vertical size differences between adjacent characters, differences in the upper and lower boundaries, and differences in the ordinate of the center point. (See reference) Figure 14 As shown in (c), the vertical alignment of text is used to describe the differences between individual characters in a column of text in the horizontal direction. For example, the vertical alignment of text can be determined by the differences in size between adjacent characters in the horizontal direction, the differences in the left boundary, the differences in the right boundary, and the differences in the horizontal coordinate of the center point.

[0238] For example, the horizontal alignment of text can be quantified using formula (1), as follows:

[0239]

[0240] In formula (1):

[0241] Rx represents the horizontal alignment of a line of text. The larger the Rx, the higher the horizontal alignment of the line of text.

[0242] y i Represents the ordinate of the i-th character in a line of text;

[0243] y i-1 Represents the ordinate of the (i-1)th character in a line of text;

[0244] h i This represents the vertical dimension of the i-th character in a line of text;

[0245] h i-1 This represents the vertical dimension of the (i-1)th character in a line of text.

[0246] n represents the number of words in a line of text.

[0247] In formula (1), the horizontal alignment of a line of text can be calculated by the average difference between adjacent characters in the line of text.

[0248] For example, the vertical alignment of text can be quantified using formula (2), as follows:

[0249]

[0250] In formula (2):

[0251] Ry represents the vertical alignment of a column of text. The larger Ry is, the higher the vertical alignment of the column of text.

[0252] L i The x-coordinate represents the left boundary of the i-th character in a text column;

[0253] L i-1 The x-coordinate of the left boundary of the (i-1)th character in a text column;

[0254] R i The x-coordinate represents the right boundary of the i-th character in a text column;

[0255] R i-1 The x-coordinate of the right boundary of the (i-1)th character in a text column;

[0256] C i The x-coordinate of the center point of the i-th character in a text column;

[0257] C i-1 The x-coordinate of the center point of the (i-1)th character in a text column;

[0258] h i This represents the vertical dimension of the i-th character in a column of text;

[0259] h i-1 This represents the vertical dimension of the (i-1)th character in a column of text.

[0260] m represents the number of words in a text column.

[0261] In formula (2), the vertical alignment of a column of text can be calculated by the mean of the differences between adjacent characters in the column.

[0262] In some embodiments, the dividing axis determined based on the first information can be used to determine the layout direction of the text block.

[0263] In this embodiment, the text block layout direction is consistent with the general reading order. A horizontally formatted text block is defined as a text block whose general reading order is to be read line by line from left to right, then from top to bottom, starting with a new line. A vertically formatted text block is defined as a text block whose general reading order is to be read column by column from top to bottom, then from left to right, starting with a new column. Horizontally and vertically formatted text blocks may be nested; that is, a horizontally formatted text block may contain a smaller vertically formatted text block. Similarly, a vertically formatted text block may contain a smaller horizontally formatted text block. Of course, due to different segmentation granularities, a horizontally formatted text block may also contain a smaller horizontally formatted text block. Similarly, a vertically formatted text block may contain a smaller vertically formatted text block.

[0264] To make it easier to understand, the following example uses the first text block of the parent node and the second text block of its child node as an illustration.

[0265] refer to Figure 15 In (a) of the figure, the general reading order of the first text block B1 within the solid line box is to read it line by line from left to right, and then read it line by line from top to bottom. Therefore, the layout direction of the first text block B1 within the solid line box is horizontal.

[0266] refer to Figure 15 In diagram (b), the general reading order of the first text block B1 within the solid line box is to read it line by line from left to right, and then from top to bottom, so the layout direction of the first text block B1 within the solid line box is horizontal. The general reading order of the second text block B2 is to read it column by column from top to bottom, and then from left to right, so the layout direction of the second text block B2 is vertical. That is, the horizontally laid-out first text block B1 contains a vertically laid-out second text block B2.

[0267] refer to Figure 15 In (c) of the figure, the general reading order of the first text block B1 within the solid line box is to read it from top to bottom column by column, and then from left to right column by column. Therefore, the layout direction of the first text block B1 within the solid line box is vertical layout.

[0268] refer to Figure 15 In diagram (d), the general reading order for the first text block B1 within the solid line box is to read it column by column from top to bottom, then from left to right, changing columns. Therefore, the layout direction of the first text block B1 within the solid line box is vertical. The general reading order for the second text block B2 is to read it line by line from left to right, then from top to bottom, changing lines. Therefore, the layout direction of the second text block B2 is horizontal. That is, the vertically laid-out first text block B1 contains a horizontally laid-out second text block B2.

[0269] In some embodiments, if the dividing axes determined according to the first information are all horizontal dividing axes (i.e., extending horizontally and penetrating the first text block) or all vertical dividing axes (i.e., extending vertically and penetrating the first text block), then the first text block can be directly divided.

[0270] For example, if the dividing axes determined based on the first information are all horizontal dividing axes, then the layout direction of the first text block can be considered as horizontal, thus enabling the first text block to be divided with horizontal dividing axes.

[0271] For example, if the dividing axes determined by the first information are all vertical dividing axes, then the layout direction of the first text block can be considered as vertical, thus the first text block can be divided by the vertical dividing axis.

[0272] In other embodiments, if the dividing axes determined based on the first information include both horizontal and vertical dividing axes, then the layout direction of the first text block needs to be precisely detected to determine the dividing axis used for actual division, in order to avoid inappropriate division leading to an incorrect reading order. Of course, if the dividing axes determined based on the first information are all horizontal or all vertical dividing axes, the layout direction of the first text block can also be precisely detected to filter out inappropriate dividing axes, thus avoiding inappropriate division leading to an incorrect reading order.

[0273] In some embodiments, taking the segmentation of the first text block as an example, step S231 may specifically include:

[0274] Get the first text block;

[0275] At least one candidate segmentation axis is determined based on the fourth information, wherein the candidate segmentation axis passes through the first text block;

[0276] The layout direction of the first text block is determined based on at least one candidate dividing axis, wherein the layout direction of the first text block is either horizontal or vertical.

[0277] The first text block is segmented based on the first segmentation axis among at least one candidate segmentation axis to obtain multiple second text blocks, wherein the extension direction of the first segmentation axis is the same as the layout direction of the first text block.

[0278] In other words, when segmenting the first text block, at least one candidate segmentation axis is first detected within the first text block. The layout direction of the first text block is determined based on this candidate segmentation axis. Then, based on the layout direction of the first text block, a first segmentation axis for actual segmentation is determined from the at least one candidate segmentation axis to segment the first text block. This makes the segmented text blocks more consistent with the general reading order and avoids errors in the sorting of text blocks.

[0279] It should be noted that the extension direction of the first dividing axis mentioned above is the same as the layout direction of the first text block. This can be understood as follows: if the layout direction of the first text block is horizontal, then the first dividing axis extends horizontally; if the layout direction of the first text block is vertical, then the first dividing axis extends vertically.

[0280] In some embodiments, the second text block is laid out horizontally or vertically.

[0281] In this embodiment, the method for determining at least one candidate segmentation axis based on the fourth information can be the same as the method for determining the segmentation axis based on the first information described above. The difference is that the candidate segmentation axes determined here may not all be used to segment the first text block; the segmentation axes actually used to segment the first text block are selected from at least one candidate segmentation axis. For details, please refer to... Figures 10 to 13 For the sake of brevity, only a brief description is provided here.

[0282] In some embodiments, the fourth information is the same as the first information. For example, the fourth information includes at least one of the following: visually salient features in the first interface (refer to...). Figure 10 ), the horizontal alignment of the text (reference) Figure 12 The vertical alignment of the text (see reference). Figure 12 The mapping relationship between the semantic information and spatial information of text (see reference). Figure 11 ), text layout density (reference) Figure 13 ).

[0283] In some embodiments, the fourth information includes the text layout density, which can be used to determine the candidate segmentation axis and the layout direction of the first text block. Specifically, the segmentation process of the first text block in step S231 can be as follows:

[0284] Obtain the adjacency matrix of the first text block, which includes the vertical or horizontal distance between adjacent words in the first text block;

[0285] The position in the first text block corresponding to the maximum value in the adjacency matrix is ​​determined as the candidate segmentation axis;

[0286] The layout direction of the first text block is determined based on at least one candidate dividing axis;

[0287] The first dividing axis for dividing the first text block is determined based on the layout direction of the first text block;

[0288] Cluster the words on both sides of the first segmentation axis to obtain multiple second text blocks, where the distance between adjacent words in the second text blocks in the vertical or horizontal direction is less than the maximum value in the adjacency matrix.

[0289] For details on segmenting the first text block based on the adjacency matrix, please refer to [link / reference]. Figure 13 For the sake of brevity, the relevant descriptions will not be repeated here.

[0290] In some embodiments, step S231 may further include segmenting the second text block, and the specific process may be as follows:

[0291] The position in the second text block that corresponds to the second maximum value in the adjacent matrix is ​​determined as the second dividing axis, and the second dividing axis runs through the second text block.

[0292] The second text block is segmented based on the second segmentation axis to obtain multiple third text blocks, wherein the distance between adjacent words in the third text block in the vertical or horizontal direction is less than the second maximum value in the adjacency matrix.

[0293] For information on multi-round segmentation based on the adjacency matrix, please refer to [link / reference]. Figure 13 For the sake of brevity, the relevant descriptions will not be repeated here.

[0294] In some embodiments, the ratio of the second-highest value to the vertical height of the smallest character within the first text block is greater than or equal to a preset threshold. For example, this preset threshold is greater than 0.

[0295] In some embodiments, while determining at least one candidate segmentation axis, the candidate segmentation axis may be marked or positioned within a first text block.

[0296] The following describes in detail the method provided in this application for determining the layout direction based on at least one candidate dividing axis (or potential dividing axis) with specific examples.

[0297] Figure 16 This shows a schematic diagram of a page related to weather information, in which Figure 16 (a) shows one or more vertical candidate segmentation axes detected for the first text block B1. Figure 16 (b) shows one or more lateral candidate segmentation axes detected for the first text block B1.

[0298] Figure 17 This illustrates a page diagram related to trip information, in which... Figure 17 (a) shows one or more lateral candidate segmentation axes detected for the first text block B1. Figure 17 (b) shows one or more vertical candidate segmentation axes detected for the first text block B1.

[0299] Figure 18 This diagram illustrates a table-like layout, in which... Figure 18 (a) shows one or more lateral candidate segmentation axes detected for the first text block B1. Figure 18 (b) shows one or more vertical candidate segmentation axes detected for the first text block B1.

[0300] It should be noted that the position of the dividing axis shown in the accompanying drawings of the embodiments of this application is merely illustrative, and the area range of the text block is also merely illustrative and should not be construed as a limitation of this application.

[0301] In this embodiment of the application, the process of determining the layout direction of the first text block based on at least one candidate segmentation axis is as follows:

[0302] S1, pre-segment the first text block B1 based on one or more detected horizontal candidate segmentation axes to obtain multiple line text regions.

[0303] For example Figure 16 As shown in (b), after pre-segmenting the first text block B1 based on multiple horizontal candidate segmentation axes, line text regions C1, C2, C3, C4, C5, C6, and C7 can be obtained. For example... Figure 17 (a) and Figure 18 As shown in (a), after pre-segmenting the first text block B1 based on multiple horizontal candidate segmentation axes, line text areas C1, C2, C3, C4, C5, and C6 can be obtained.

[0304] S2, determine the interline layout similarity of the first text block B1 based on the fifth information.

[0305] The fifth piece of information may include at least one of the following: differences in alignment (or degree of alignment) of line text intervals, differences in line width of line text intervals, differences in the number of texts in line text intervals, differences in the size of text in the same column of line text intervals, and differences in line spacing of line text intervals.

[0306] For example, the alignment of a line text area refers to the alignment of the text within that area, such as left alignment, right alignment, or center alignment. The line width of a line text area refers to its horizontal dimension. The number of text elements in a line text area can refer to the number of words or phrases included within that area. Text in the same column within a line text area refers to text that corresponds vertically across different line text areas.

[0307] In this embodiment of the application, the layout similarity between any two line text areas can be obtained based on the fifth information. For example, in S2, the interline layout similarity of the first text block B1 can be determined based on the layout similarity between adjacent line text areas.

[0308] by Figure 16 Taking (b) as an example, the interline layout similarity of the first text block B1 can be determined based on the layout similarity between text areas C1 and C2, between text areas C2 and C3, between text areas C3 and C4, between text areas C4 and C5, between text areas C5 and C6, and between text areas C6 and C7.

[0309] For example, if each line text area includes a line of text, the interline layout similarity of the first text block B1 can be quantified by formula (3), as follows:

[0310]

[0311] In formula (3):

[0312] h i,j This represents the vertical dimension of the character in the i-th row and j-th column;

[0313] hi,j-1 This represents the vertical dimension of the character in the i-th row and j-1-th column;

[0314] T i,j+1 The ordinate represents the upper boundary of the character in the i-th row and j+1-th column;

[0315] B i,j The ordinate represents the lower boundary of the character in the i-th row and j-th column;

[0316] T i,j The ordinate represents the upper boundary of the character in the i-th row and j-th column;

[0317] B i,j-1 The ordinate represents the lower boundary of the character in the i-th row and j-1-th column;

[0318] m represents the row number;

[0319] n represents the column number.

[0320] In formula (3), the interline layout similarity of the first text block B1 can be calculated by the mean of the differences between adjacent text areas, where the differences between adjacent text areas are obtained by calculating the differences between texts in the same column in adjacent text areas.

[0321] For example, refer to Figure 17 In (a), if each line of text area includes multiple lines of text, the interline layout similarity of the first text block B1 can be quantified by formula (4), as follows:

[0322]

[0323] In formula (4):

[0324] h q,i,j This represents the vertical dimension of the character in the i-th row and j-th column of the q-th line of the text area;

[0325] h q,i,j-1 This represents the vertical dimension of the character in the i-th row and j-1-th column of the q-th line of the text area;

[0326] T q,i,j+1 The ordinate represents the upper boundary of the character in the (j+1)th column of the (i)th row of the (q)th line of the text area;

[0327] B q,i,j The ordinate represents the lower boundary of the character in the i-th row and j-th column of the q-th line of the text area;

[0328] T q,i,j The ordinate represents the upper boundary of the character in the i-th row and j-th column of the q-th line of the text area;

[0329] B q,i,j-1The ordinate represents the lower boundary of the character in the i-th row and j-1-th column of the q-th line of the text area;

[0330] m represents the number of line text areas;

[0331] n represents the number of columns in a single line of text (i.e., the number of columns in a single line of text);

[0332] p represents the number of lines in a single text area (i.e., the number of lines in a single text area).

[0333] In formula (4), the interline layout similarity of the first text block B1 can be calculated by the mean of the differences between adjacent text areas. The differences between adjacent text areas are obtained by calculating the differences between corresponding lines of text in adjacent text areas. The differences between corresponding lines of text in adjacent text areas include: the difference between the first line of text in one text area and the first line of text in another text area, the difference between the second line of text in one text area and the second line of text in another text area, and so on.

[0334] S3, pre-segment the first text block B1 based on one or more detected vertical candidate segmentation axes to obtain multiple column text areas.

[0335] For example Figure 16 As shown in (a), after pre-segmenting the first text block B1 based on multiple vertical candidate segmentation axes, column text areas P1, P2, P3, P4, and P5 can be obtained. For example... Figure 17 (b) or Figure 18 As shown in (b), the first text block B1 can be pre-segmented based on multiple vertical candidate segmentation axes to obtain column text areas P1, P2, P3, and P4.

[0336] S4. Determine the inter-column layout similarity of the first text block B1 based on the sixth information.

[0337] The sixth piece of information includes at least one of the following: differences in alignment of column text intervals, differences in column width of column text intervals, differences in the number of texts in column text intervals, differences in the size of text in the same row of column text intervals, and differences in column spacing of column text intervals.

[0338] For example, the alignment of a column text area refers to the alignment of the text within that area, such as top alignment, bottom alignment, or center alignment. The column width of a column text area refers to its horizontal dimension. The number of text items in a column text area can refer to the number of words or phrases included within that area. Text in the same row within a column text area refers to the corresponding text in different column text areas horizontally.

[0339] In this embodiment of the application, the layout similarity between any two column text areas can be obtained based on the sixth information. For example, in S4, the inter-column layout similarity of the first text block B1 can be determined based on the layout similarity between adjacent column text areas.

[0340] by Figure 16 Taking (a) as an example, the inter-column layout similarity of the first text block B1 can be determined based on the layout similarity between text areas P1 and P2, the layout similarity between text areas P2 and P3, the layout similarity between text areas P3 and P4, and the layout similarity between text areas P4 and P5.

[0341] For example, if each column text area includes a column of text, the inter-column layout similarity of the first text block B1 can be quantified by formula (5), as follows:

[0342]

[0343] In formula (5):

[0344] h i,j This represents the vertical dimension of the character in the i-th row and j-th column;

[0345] h i,j-1 This represents the vertical dimension of the character in the i-th row and j-1-th column;

[0346] L i,j+1 The x-coordinate of the left boundary of the character in the i-th row and j+1-th column;

[0347] R i,j The x-coordinate of the right boundary of the character in the i-th row and j-th column;

[0348] L i,j The x-coordinate of the left boundary of the character in the i-th row and j-th column;

[0349] R i,j-1 The x-coordinate of the right boundary of the character in the i-th row and j-1-th column;

[0350] m represents the row number;

[0351] n represents the column number.

[0352] In formula (5), the inter-column layout similarity of the first text block B1 can be calculated by the mean of the differences between adjacent column text areas, where the differences between adjacent column text areas are obtained by calculating the differences between the text in the same row in adjacent column text areas.

[0353] For example, if each column text area includes multiple columns of text, the inter-column layout similarity of the first text block B1 can be quantified by formula (6), as follows:

[0354]

[0355] In formula (6):

[0356] h q,i,j This represents the vertical dimension of the character in the i-th row and j-th column of the q-th column text area;

[0357] h q,i,j-1 This represents the vertical dimension of the character in the i-th row and j-1-th column of the q-th column text area;

[0358] L i,j+1 The x-coordinate represents the left boundary of the character in the i-th row and j+1-th column of the q-th column text area;

[0359] R q,i,j The x-coordinate represents the right boundary of the character in the i-th row and j-th column of the q-th column text area;

[0360] L q,i,j The x-coordinate represents the left boundary of the character in the i-th row and j-th column of the q-th column text area;

[0361] R q,i,j-1 The x-coordinate represents the right boundary of the character in the i-th row and j-1-th column of the q-th column text area;

[0362] m represents the number of rows in a column of text (that is, the number of rows in a column of text);

[0363] n represents the number of text sections in the column;

[0364] k represents the number of columns in a text area (i.e., the number of columns in a text area).

[0365] In formula (6), the inter-column layout similarity of the first text block B1 is calculated by the mean of the differences between adjacent column text areas. The differences between adjacent column text areas are obtained by calculating the differences between corresponding column texts in adjacent column text areas. The differences between corresponding column texts in adjacent column text areas include: the difference between the first column text of one column text area and the first column text of another column text area, the difference between the second column text of one column text area and the second column text of another column text area, ..., and so on.

[0366] It should be noted that the aforementioned formulas (1) to (6) are independent formulas, and the meaning of the parameters included in each formula is defined only for each formula. The parameter symbols and definitions in a certain formula do not limit other formulas.

[0367] S5. Determine the layout direction of the first text block B1 based on the interline layout similarity and the inter-column layout similarity of the first text block B1.

[0368] Specifically, if the interline layout similarity of the first text block B1 is greater than the inter-column layout similarity of the first text block B1 (e.g., Sx > Sy), the layout direction of the first text block is determined to be horizontal. If the inter-column layout similarity of the first text block B1 is greater than the interline layout similarity of the first text block B1 (e.g., Sy > Sx), the layout direction of the first text block is determined to be vertical.

[0369] For example, according to Figure 16 As can be seen in (a), the differences between the various column text areas in terms of alignment, column width, number of text characters, size of text in the same position, and column spacing are relatively small. According to Figure 16 As can be seen from (b), at least the line text areas C5 and C6 differ significantly from other line text areas in terms of alignment, size of text in tandem, and line spacing. Therefore, the inter-column layout similarity of the first text block B1 is greater than the inter-line layout similarity, and the first text block B1 is vertically formatted.

[0370] For example, according to Figure 17 As can be seen in (a), the differences between the various text areas in terms of alignment, line width, number of text characters, size of text in tandem, and line spacing are relatively small. According to Figure 17 As can be seen from (b) in the figure, at least the column text area P4 is significantly different from other column text areas in terms of column width and number of texts. Therefore, the interline layout similarity of the first text block B1 is greater than the inter-column layout similarity, and the first text block B1 is horizontally laid out.

[0371] For example, according to Figure 18 As can be seen in (a), the differences between the various text areas in terms of alignment, line width, number of text characters, size of text in tandem, and line spacing are relatively small. According to Figure 18 As can be seen from (b), the text areas in each column differ significantly, at least in terms of column width. Therefore, the interline layout similarity of the first text block B1 is greater than the inter-column layout similarity, and the first text block B1 is horizontally formatted.

[0372] In the above embodiments, at least one candidate segmentation axis includes one or more horizontal candidate segmentation axes and one or more vertical candidate segmentation axes. Therefore, the interline layout similarity and inter-column layout similarity of the first text block B1 can be obtained through the above steps S1 to S5.

[0373] In some other embodiments, if at least one candidate segmentation axis is a horizontal candidate segmentation axis, the interline layout similarity of the first text block can be obtained based on the above steps S1 and S2. When the interline layout similarity of the first text block is greater than a first preset value, the layout direction of the first text block is determined to be horizontal. Otherwise, the first text block is considered unsuitable for further segmentation.

[0374] In some other embodiments, if at least one candidate segmentation axis is a vertical candidate segmentation axis, the inter-column layout similarity of the first text block can be obtained based on the above steps S3 and S4. When the inter-column layout similarity of the first text block is greater than a second preset value, the layout direction of the first text block is determined to be vertical. Otherwise, the first text block is considered unsuitable for further segmentation.

[0375] In some other embodiments, if no vertical dividing axis is detected that runs through the first text block, the layout direction of the first text block can be assumed to be horizontal.

[0376] In some embodiments, after determining the layout direction of the first text block, the electronic device can mark its layout direction for sorting text blocks at the same level when constructing a tree structure.

[0377] In some embodiments, after determining the layout direction of the first text block B1, the first dividing axis can be determined from at least one candidate dividing axis based on the layout direction of the first text block B1.

[0378] Specifically, when the layout direction of the first text block B1 is horizontal, the first dividing axis is the aforementioned horizontal candidate dividing axis. When the layout direction of the first text block B1 is vertical, the first dividing axis is the aforementioned vertical candidate dividing axis.

[0379] Based on the selected dividing axes, the first text block can be divided into smaller text blocks to achieve a column-based effect.

[0380] For example, refer to Figure 16 The first text block B1 is vertically formatted, therefore the vertical candidate segmentation axis is the segmentation axis used to actually segment the first text block B1, i.e., the first segmentation axis. Segmenting the first text block B1 based on the selected first segmentation axis can yield smaller second text blocks, such as the text blocks formed by column text areas P1, P2, P3, and P4 respectively.

[0381] For example, refer to Figure 17 and Figure 18 The first text block B1 is horizontally formatted, therefore the horizontal candidate segmentation axis is the segmentation axis used to actually segment the first text block B1, i.e., the first segmentation axis. Segmenting the first text block B1 based on the selected first segmentation axis can yield second text blocks with smaller granularities, such as the text blocks formed by the line text areas C1, C2, C3, C4, C5, and C6 respectively.

[0382] By utilizing the layout direction of the first text block to filter out the actual dividing axes from at least one candidate dividing axis, unsuitable dividing axes can be eliminated, avoiding unreasonable dividing methods. For example... Figure 18While the horizontally aligned table-like layout shown can identify candidate dividing axes formed by strict left-right alignment and large horizontal gaps between the text on both sides, it cannot be simply divided vertically by columns, as this would violate the normal reading order. By filtering candidate dividing axes using the layout direction of the first text block, column dividing axes in the table-like layout can be eliminated, ensuring that the table-like layout is divided according to the normal reading order.

[0383] In some embodiments, for table-like layouts, column dividing axes can be removed as follows: If it is determined that the horizontal alignment of the text areas on both sides of a vertical candidate dividing axis is high (e.g., greater than threshold A), and the text size variation within each text area is small (e.g., less than threshold B), then the vertical candidate dividing axis can be identified as a table-like column dividing axis, and this vertical candidate dividing axis is removed when determining the first dividing axis.

[0384] In the above embodiments, the first segmentation axis used to actually segment the first text block is selected from candidate segmentation axes based on the layout direction of the first text block after the layout direction of the first text block is detected. In other embodiments, the process of determining the first segmentation axis can be decoupled from the process of detecting the layout direction of the first text block. For example, the layout direction of the first text block can be implemented using any of the layout direction detection methods described above. After determining the layout direction of the first text block, the electronic device can determine the first segmentation axis based on the layout direction using any of the aforementioned methods for determining the segmentation axis according to the first information. For example, if the layout direction of the first text block is determined to be horizontal, the first information can be used to determine the horizontal segmentation axis used to actually segment the first text block. If the layout direction of the first text block is determined to be vertical, the first information can be used to determine the vertical segmentation axis used to actually segment the first text block.

[0385] For example, the electronic device can first detect whether there is a vertical candidate dividing axis that runs through the first text block. If not, it directly determines that the layout direction of the first text block is horizontal. Based on the horizontal layout direction of the first text block, the electronic device then determines the horizontal dividing axis used to divide the first text block according to the first information.

[0386] For example, the electronic device can first detect whether there is a horizontal candidate dividing axis that runs through the first text block. If not, it directly determines that the layout direction of the first text block is vertical. Based on the vertical layout direction of the first text block, the electronic device then determines the vertical dividing axis used to divide the first text block according to the first information.

[0387] For example, if the electronic device detects a horizontal candidate segmentation axis and a vertical candidate segmentation axis that run through the first text block, and determines the layout direction of the first text block based on the horizontal and vertical candidate segmentation axes, the electronic device can then determine the segmentation axis used to actually segment the first text block based on the first information.

[0388] In other words, the process of detecting the layout direction of the first text block and determining the actual dividing axis both involve the step of detecting the dividing axis. The methods used to detect the dividing axis in the two steps can be the same or different.

[0389] In this embodiment of the application, there are multiple ways to obtain the first text block.

[0390] As an example, the first text block can be obtained by splitting a larger text block. The splitting method can refer to the method used to split the first text block.

[0391] As another example, the horizontal alignment of each line of text and / or the vertical alignment of each column of text in the text content can be obtained; multiple consecutive lines of text and / or multiple consecutive columns of text are identified as a first text block, wherein the horizontal alignment of the text lines is greater than a first threshold and the vertical alignment of the text columns is greater than a second threshold.

[0392] For example, text content can be scanned from beginning to end to find continuously horizontally aligned text lines or continuously vertically aligned text columns, and these consecutive text lines or columns can be identified as the first text block. In some embodiments, continuously horizontally aligned text lines can be referred to as potential lines, and continuously vertically aligned text columns as potential columns. A potential line is defined as a composition consisting of consecutive top-aligned, bottom-aligned, or center-aligned words. A potential column is defined as a combination consisting of consecutive left-aligned, right-aligned, or center-aligned words.

[0393] The above provides a detailed description of the text block division in step S231. Now, return to the reference. Figure 9 Steps S232 and S233 will be described below.

[0394] like Figure 9 As shown, after the text blocks are divided, in step S232, the hierarchical relationship between the divided text blocks is determined according to the second information.

[0395] The second information may include at least one of the following: spatial information of the text, semantic information of the text, and visual saliency features in the first interface. For a detailed description of the second information, please refer to the above introduction of the first information; for brevity, it will not be repeated here.

[0396] As an example, the second piece of information can include spatial information about the text. For instance, if the spatial information of a text block determines that one text block is located within the area of ​​another text block, then the hierarchy of the other text block is higher than that of the first text block. Similarly, if the spatial information of two text blocks determines that they are located on the same line of the first interface, then the hierarchy of the two text blocks is the same. Furthermore, if a text block is divided into smaller text blocks, then the hierarchy of the smaller text blocks is lower than that of the original text block.

[0397] As another example, the second information can include semantic information about the text. For instance, if two text blocks have opposite semantics, such as "outbound" and "return", then it can be determined that the two text blocks are at the same level.

[0398] As another example, the second piece of information can include visually salient features in the first interface, such as shadows, color blocks, and borders. For instance, if two text blocks are located in different color block areas or border areas, it can be determined that the two text blocks are at the same level. Similarly, if the background color range of one text block is larger than that of another text block, it can be determined that the first text block is at a higher level than the second.

[0399] In fact, after the aforementioned step S231, the text content has been segmented into multiple text blocks, which are mutually inclusive. In the tree structure constructed based on the segmented text blocks, each text block is a node of the tree structure. Each level of the tree structure represents a text structure, and the bottom-level leaf nodes are the original words identified in step S210.

[0400] Step S232 enables the hierarchical organization of text content.

[0401] S233, Sort the divided text blocks according to the third information.

[0402] The third piece of information includes the spatial information of the text. For example, the spatial information of the text may include the position and size of the text. The position of the text may include the coordinates of the text within the first interface and / or the line index of the text within the first interface.

[0403] In this step, the spatial information of the text can be used to determine the positional relationship between two text blocks, so that the text blocks can be sorted in an order that is more in line with general reading habits.

[0404] In this embodiment of the application, based on the hierarchical relationship between text blocks determined in step S232, the text blocks can be sorted according to the following principles:

[0405] 1) The text blocks obtained by horizontal cutting are sorted from top to bottom;

[0406] 2) The text blocks obtained by vertical cutting are sorted from left to right;

[0407] 3) Sort the upper-level structure first, then sort the lower-level structure.

[0408] In principle 1), the positional relationship between the text blocks obtained by horizontal cutting can be determined based on third information. For example, the relative positional relationship of the text blocks in the vertical direction can be determined based on the coordinates of the text blocks or the index of the text lines.

[0409] In principle 2), the positional relationship between the text blocks obtained by vertical segmentation can be determined based on the third information. For example, the relative positional relationship of the text blocks in the horizontal direction can be determined based on the coordinates of the text blocks or the index of the text lines.

[0410] Taking the aforementioned first and second text blocks as examples, when multiple second text blocks are obtained by segmenting the first text block based on a horizontal dividing axis, the multiple second text blocks are sorted from top to bottom according to their spatial information. For example, refer to... Figure 7 Text block a (an example of the first text block) is horizontally divided into text block b (an example of the second text block) and text block c (an example of the second text block). Since text block b is located above text block c, text block b is ordered first, and text block c is ordered last. For example, see reference... Figure 8 Text block d1 (an example of the first text block) is horizontally divided to obtain text blocks e1, e2, e3, e4, e5, and e6 (several examples of the second text block). Based on the spatial information of text blocks e1, e2, e3, e4, e5, and e6, text blocks e1, e2, e3, e4, e5, and e6 are sorted from top to bottom.

[0411] It can be seen that the direction in which the text blocks obtained by horizontal segmentation are sorted from top to bottom is consistent with the layout direction of the segmented text blocks, which conforms to the reading order of horizontal layout.

[0412] Taking the aforementioned first and second text blocks as examples, when multiple second text blocks are obtained by segmenting the first text block based on the vertical segmentation axis, the multiple second text blocks are sorted from left to right according to their spatial information. For example, refer to... Figure 7 Text block c (an example of the first text block) is vertically segmented to obtain text blocks g1, g2, and g3 (several examples of the second text block). Based on the spatial information of text blocks g1, g2, and g3, they are sorted from left to right. For example, see reference... Figure 8Text block c (an example of the first text block) is vertically divided to obtain text blocks d1, d2, d3, d4, and d5 (several examples of the second text block). Based on the spatial information of text blocks d1, d2, d3, d4, and d5, text blocks d1, d2, d3, d4, and d5 are sorted from left to right.

[0413] It can be seen that the direction in which the text blocks obtained by vertical segmentation are sorted from left to right is consistent with the layout direction of the segmented text blocks, which conforms to the reading order of vertical layout.

[0414] It should be noted that, Figure 7 or Figure 8 The index on the branch indicates the sorting order of the child nodes under each parent node.

[0415] Taking the aforementioned first and second text blocks as examples, the sorting priority of the first text block is higher than that of the second text block; that is, the first text block is ordered before the second text block. (See reference) Figure 8 Text block b is the upper-level structure of text blocks d1 to d5, and its order must precede that of the lower-level structure. This arrangement of text order is more in line with general reading habits.

[0416] After sorting, the sorting of the lowest-level text blocks is taken as the final result. (Reference) Figure 7 or Figure 8 If text block 'a' is the root node, then the bordered numbers marked on text block 'a' indicate the sorting order of all leaf nodes within the entire text block 'a', which is also the final word order and the default order of text selected from the first interface. It can be seen that the rearranged text order is more in line with general reading habits.

[0417] This application presents a tree structure for constructing text content, which enables precise segmentation of the original layout text and sorting according to the general reading order. In the process of constructing the tree structure, layout analysis based on text location information offers greater real-time performance compared to layout analysis that relies entirely on image information.

[0418] In some embodiments, before step S231, step S230 may further include: dividing all words in the text content into multiple text lines, each text line horizontally spanning the first interface; sorting the multiple text lines from top to bottom, and obtaining the text line index of each text line. Here, the text line index is used to divide the text content into text blocks and / or sort the text blocks. The text line index is global and can also be called the global line index.

[0419] For example, text line indexes can be used to determine whether words are on the same line or adjacent vertically, which can be used to divide text blocks. Similarly, text line indexes can be used to determine the relative positions of text blocks, which can be used to sort the text blocks.

[0420] Through steps S231 to S233 described above, a tree structure can be constructed for the text content. Each level of the tree structure represents a text block, and the bottom-level leaf nodes of the tree structure are single words or consecutive lines. Based on the tree structure, users can select text in units of text blocks on the first interface, and can flexibly select text blocks at different levels, which improves selection efficiency and meets the user's selection intentions.

[0421] The following section provides a more detailed description of the text selected by the user in step S240, with reference to the accompanying drawings.

[0422] In some embodiments, step 240 specifically includes: determining the text block that the user intends to select based on the trajectory of the text selected by the user and the tree structure, wherein the trajectory of the text selected by the user passes through the area of ​​at least one second text block, and the text block that the user intends to select includes at least one second text block; highlighting at least one second text block.

[0423] In other words, based on the tree structure, the text block selected by the user can be determined according to the trajectory of the user's selected text. When selecting text, the user can make efficient text selections in a way that conforms to the general reading flow, without the problems of interleaved selections or random selections between different structures. When there are multiple possible arrangements of text blocks, the trajectory of the user's selected text can freely express the text selection order that conforms to their intention.

[0424] Figure 19 The diagram shows a page related to the trip. For clarity, the selected text blocks are indicated only by dashed borders, with each closed dashed border representing a text block.

[0425] refer to Figure 19 Based on the pre-constructed tree structure, users can select text using a selection cursor (including its start and end positions). Correspondingly, the electronic device can determine the text block the user intends to select based on the trajectory of the cursor. This pre-constructed tree structure, along with the start and end positions of the selection cursor, allows users to select text in a typical reading order.

[0426] In one example, a user can use the cursor to select one outbound flight ticket and multiple return flight tickets on the first screen. For example... Figure 19 In (a), the user can select a one-way flight ticket by dragging the cursor. When the user drags the cursor to the right, as shown in (a), the user can select a one-way flight ticket by dragging the cursor to the right. Figure 19As shown in (b), another return flight ticket can be selected. When the user continues to drag the cursor down, as shown... Figure 19 As shown in (c), you can continue to select the second return flight information without selecting the second outbound flight information.

[0427] In another example, a user can select multiple outbound or return trip records using the cursor on the first screen. For example... Figure 19 In (a), the user can select a one-way flight ticket by dragging the cursor. When the user drags the cursor down, as shown... Figure 19 As shown in (c), you can continue to select one or more outbound flight tickets without selecting the return flight tickets.

[0428] When there are multiple possible arrangements of structures, the cursor trajectory of the selected text can freely express the text selection order that matches the user's intention. When a user wants to select multiple flight times and departure points, they don't need to make multiple selections, resulting in high selection efficiency.

[0429] Figure 20 The diagram shows a page related to the trip. For clarity, the selected text blocks are indicated by dashed borders and the smaller text blocks within each selected text block are indicated by solid boxes. Each closed dashed border and solid box represents a text block.

[0430] refer to Figure 20 Based on the pre-constructed tree structure, users can select text using smudge gestures. Correspondingly, the electronic device can determine the text block the user intends to select based on the trajectory of the smudges.

[0431] For example, refer to Figure 20 In (a), users can horizontally select one outbound flight ticket and one return flight ticket.

[0432] For example, refer to Figure 20 In section (b), users can vertically select multiple outbound flight tickets. Alternatively, users can vertically select multiple return flight tickets.

[0433] For example, refer to Figure 20 In (c), users can first paint horizontally and then vertically to select one outbound flight ticket and multiple return flight tickets.

[0434] For example, refer to Figure 20 In (d), users can select multiple outbound flight tickets and multiple return flight tickets through discontinuous swiping operations.

[0435] When there are multiple possible arrangements of structures, the user's smudge path can express the hierarchical correspondence of their intended selection. Compared to smudge text lines, supporting fuzzy smudges at the structure level improves interaction efficiency and has a higher fault tolerance.

[0436] Figure 21 The diagram shows a page related to the transfer record. For clarity, the selected text block and the text block above the selected text block are indicated only by dashed and solid borders. Each closed dashed and solid border represents a text block.

[0437] refer to Figure 21 Based on the pre-constructed tree structure, users can select text based on clicks. Accordingly, the electronic device can determine the text block the user intends to select based on the user's click trajectory (including at least one click location).

[0438] For example, refer to Figure 21 In (a), the user can click to select the text block of the lowest level node, that is, the text block of the leaf node in the tree structure, thereby selecting the balance information.

[0439] For example, refer to Figure 21 In (b), users can click to select the text block of the node above the lowest level (i.e., the second to last level) to select a transfer record without a date.

[0440] For example, refer to Figure 21 In (c), the user can click to select the text block of the third-to-last node, thereby selecting a transfer record with a date.

[0441] For example, refer to Figure 21 In (d), users can click to select the text block of the fourth-to-last node, thereby selecting the entire month's transfer records.

[0442] In click-based text selection, users can select text at a given location by clicking on it, or select the lowest-level text block corresponding to a blank space by clicking on a blank space. This allows users to select a line of plain text or an entire entry with a single click, improving interaction efficiency.

[0443] In some embodiments, before a user makes a selection, the electronic device may display the outlines of each text block on a first interface to help the user determine the location of the text block to be selected.

[0444] In some embodiments, when a user clicks to select text, the electronic device may display the outline of the lowest-level text block corresponding to the clicked location so that the user can confirm whether it is the text block intended to be selected.

[0445] In some embodiments, the click position of the user's selected word is used to determine the level of the selected text block, so as to determine whether the selected word is a single word, a low-level text block, or a larger high-level text block.

[0446] For example, when a user clicks to select text, the electronic device can determine the lowest-level text block corresponding to the user's click location based on the user's click location and the boundary information of the text blocks corresponding to each node in the tree structure, and determine that the lowest-level text block is the text block that the user intends to select.

[0447] In this embodiment, the boundary of a text block at least encloses the text of the corresponding text block. In some embodiments, the boundary of a text block may also include a region without text. The boundaries of the child nodes under the parent node do not overlap. For example, the boundary of a text block can be a rectangular border that encloses the corresponding text.

[0448] In some embodiments, reference Figure 22 As shown, when constructing a tree structure, an undirected graph can be built between nodes at the same level, where directly adjacent text blocks can be bidirectionally connected. The connectivity between nodes in the tree structure represents the possible paths for selecting text, while the user's dragging of the cursor, scribbling, or clicking represents the path selection. In this way, the tree structure can provide users with more selection sequences, better satisfying their selection intentions.

[0449] Although the text is rearranged to conform to the general reading order in the tree structure construction, users may need to select text in different orders to meet diverse information dissemination needs within the same page. Therefore, a single order obtained solely from rearranging the text cannot satisfy users' multiple text selection intentions. In this embodiment, by constructing an undirected graph among nodes at the same level, it is possible to infer the text blocks that the user intends to select and the order among the selected texts based on the trajectory of the user's text selection.

[0450] like Figure 23As shown in (a), after a user selects the first word A in the first line of text block Q1, if they drag the selection cursor to the right, it is inferred that they want to select the second word B to the right of the first word in the first line. If the user selects the first word A in the first line and then drags the selection cursor down, it is inferred that they want to select the first word C in the second line below A. If the user selects A and C and continues to drag the cursor to the right, it is inferred that they want to select B and D to the right of A and C (that is, the first two lines of text block Q1). When the user continues to drag the selection cursor to the right, it is inferred that the word they intend to select has left the current word's parent node and they intend to continue selecting child nodes (words) under other parent nodes. Therefore, when the user's intention to leave the current parent node and continue selecting words in other parent nodes at the same level is detected, the word selected by the user under the current parent node is recorded as a template, and this template (including the initial line order of words in the text block, etc.) is used to quickly select words under the new parent node.

[0451] exist Figure 23 In step (c), the user has already selected the first two lines of words in text block Q1. Therefore, recording the first two lines of words in this parent node serves as a selection template, which is then used in the subsequent word selection logic of other parent nodes at the same level. Thus, dragging the cursor to the right will directly select the first two lines of text block Q2 on the right, and dragging it down will select the first two lines of text block Q3 below, and so on.

[0452] Depend on Figure 22 As shown in the undirected graph structure, this free word selection method based on the user's word selection trajectory relies on an undirected graph structure, which is formed based on the previously constructed tree hierarchy. In this undirected graph, bidirectional paths are established between child nodes (i.e., sibling nodes) of the same parent node at the same level in the original tree hierarchy. This bidirectional path means that the user can select words from the word corresponding to a child node to the word corresponding to its sibling node in any direction, and can also select words from the word under a parent node to the word under other parent nodes that are sibling nodes of that parent node.

[0453] Therefore, in some embodiments, in order to further improve selection efficiency, the electronic device can also determine the text block that the user intends to select in the following way: when the user's selection area is detected to switch from a child node (e.g., a leaf node) to a parent node, the selected child node in this layer is marked as a template; when the user's selection area is detected to switch to a sibling node of the parent node, the child nodes (e.g., leaf nodes) under the sibling node can be mapped and selected according to the template.

[0454] Taking the aforementioned first and second text blocks as examples, when the electronic device detects that the trajectory of the user's selected text changes from the area of ​​the second text block to the area of ​​the first text block, the hierarchy and sorting information of the second text block selected in the first text block in the tree structure are marked as a template; when the electronic device detects that the trajectory of the user's selected text changes from the area of ​​the first text block to the area of ​​the sibling node of the first text block, the text block that the user intends to select is determined according to the template, including the text blocks corresponding to the child nodes under the sibling node.

[0455] In other words, an electronic device can map and select the text block corresponding to the child node of a sibling node of a parent node based on the user's selection of child nodes under that parent node; that is, it can map and select the cousin node of the selected child node. For example, the child node and its cousin node are at the same level, and the sorting or position of the child node under its parent node is the same as the sorting and position of the selected cousin node under its parent node. The following is combined with... Figure 23 and Figure 24 To be further described.

[0456] refer to Figure 23 In (a), text blocks Q1, Q2, Q3, and Q4 are sibling nodes with similar layouts, and each text block has multiple child nodes. For example... Figure 23 In (b) and (c), the user can drag the cursor to select multiple child nodes under text block Q1. For example... Figure 23 As shown in (d), when the user drags the cursor from the area of ​​text block Q1 to the area of ​​text block Q2, the electronic device infers that the user intends to select the child node in text block Q2 that corresponds to the position of the selected child node in text block Q1. At this time, the electronic device automatically selects the corresponding text block in text block Q2, without requiring the user to select them one by one. Figure 23 As shown in (e), when the user drags the cursor from the area of ​​text block Q2 to the area of ​​text block Q3, the electronic device infers that the user intends to select the child node in text block Q3 that corresponds to the position of the selected child node in text block Q1. Similarly, at this time, the electronic device automatically selects the corresponding text block in text block Q3 without requiring the user to select them one by one.

[0457] refer to Figure 24 In (a), text blocks Q1 and Q2 are cousin nodes, and each cousin node has multiple child nodes. For example... Figure 24 In (b) and (c), the user can select multiple child nodes under text block Q1 through a smudge operation (e.g., selecting a transfer record with a date). Figure 24As shown in (d), when the user's smearing trajectory switches from the parent node region of text block Q1 to the parent node region of text block Q2, the electronic device infers that the user intends to select the child node in text block Q2 that corresponds to the position of the selected child node in text block Q1. At this time, the electronic device automatically selects the corresponding text block in text block Q2 (i.e., another transfer record with a date) without requiring the user to select them one by one.

[0458] In some embodiments, the electronic device highlights the text block selected by the mapping (i.e., the text block corresponding to the child node under the sibling node, i.e., the cousin node of the child node).

[0459] In some embodiments, the visual salience features of the mapped text block differ from those of the second text block. For example, the background color of the mapped text block is different from that of the second text block; or, the mapped text block has a blinking effect; or the mapped text block displays a structural outline.

[0460] In some embodiments, when the mapped text block is pasted and displayed on the second interface, its visual salience in the second interface may be different from that of the second text block in the second interface, in order to prompt the user.

[0461] In some embodiments, the second interface may include a selection control, allowing users to choose whether to retain or delete the mapped text block content.

[0462] The text selection process has been explained in detail above with reference to the accompanying diagrams. Now, return to the references. Figure 2 In some embodiments, method 200 further includes:

[0463] S250, in response to the user's paste operation, displays the content of at least one second text block on the second interface according to the tree structure.

[0464] In this step, the electronic device displays the content of at least one second text block on the second interface based on the hierarchical information and text sorting information provided by the tree structure.

[0465] It is understandable that in this step, when the user selects a text block from another parent node, this step will also paste and display the content of the selected text block on the second interface.

[0466] After selecting words in a manner consistent with reading order, users often need to paste the text as formatted or plain text. In this embodiment, the tree structure constructed for the text content can provide users with a general reading order. Therefore, when the user pastes the selected text block content onto the second interface, the selected text block will also display relevant information in an order consistent with reading habits.

[0467] In one example, the text blocks pasted to the second interface are displayed in lines according to the smallest hierarchical text block structure. That is, each leaf node of the selected content is displayed on a separate line, and the content of different leaf nodes is displayed on different lines. Based on the order of the leaf nodes in the tree structure, the content can be displayed on the second interface in a way that conforms to the reading order.

[0468] For example, refer to Figure 25 As shown, Figure 25 The hierarchical structure of the text blocks shown is similar to Figure 7 The text blocks shown have the same hierarchical structure. (The remaining text appears to be incomplete and possibly contains errors.) Figure 25 When the text block content shown is pasted and displayed on the second interface, the display format can be as shown in Paste Result 1, with all leaf nodes arranged according to... Figure 7 The order in the hierarchical structure shown is displayed in rows.

[0469] For example, refer to Figure 26 As shown, according to Figure 26 The hierarchical structure of the text block shown can be displayed as shown in Paste Result 1 when pasted onto the second interface, with all leaf nodes arranged according to... Figure 26 The order in the hierarchical structure shown is displayed in rows.

[0470] For example, refer to Figure 27 As shown, Figure 27 The hierarchical structure of the text blocks shown is similar to Figure 8 The text blocks shown have the same hierarchical structure. (The remaining text appears to be incomplete and possibly contains errors.) Figure 27 When the text block content shown is pasted and displayed on the second interface, the display format can be as shown in Paste Result 1, with all leaf nodes arranged according to... Figure 8 The order in the hierarchical structure shown is displayed in rows.

[0471] In another example, when the text block content pasted to the second interface is displayed, child nodes with the same parent node are displayed on the same line, while child nodes with different parent nodes are displayed on separate lines.

[0472] When reading, people generally read text that is close in position together because these texts appear simultaneously in a small reading hotspot. Child nodes with the same parent node are usually quite close in position, so text blocks with the same parent node can be grouped together on a single line. Here, the degree of text proximity can be determined by the granularity of the text blocks segmented in the preceding steps. Generally, the finer the granularity of the text blocks, the higher the degree of proximity between the texts.

[0473] In some embodiments, nodes belonging to the same lower level (e.g., within the third level from the bottom) under the same parent node are displayed in the same row.

[0474] In some embodiments, different text blocks are displayed on separate lines.

[0475] For example, refer to Figure 25 and Figure 7 ,according to Figure 7 As shown in the hierarchy, text blocks f1, f2, f3, f4, f5, and f6 share the same parent node d, therefore they can be displayed on the same line. Text blocks g1, g2, and g3 share the same parent node c, therefore they can also be displayed on the same line. According to... Figure 7 It can be seen that text blocks f1 / f2 / f3 / f4 / f5 / f6 / g1 / g2 / g3 have different parent nodes from text block e, therefore text block c is displayed on a separate line. For example, Figure 7 The text block shown can be displayed in the second interface in the following way: Figure 25 The pasting result shown is 2.

[0476] For example, refer to Figure 26 ,according to Figure 26 As shown in the hierarchy, text blocks d1 and d2 share the same parent node b, therefore they can be displayed on the same line. Text blocks e1, e2, e3, and e4 share the same parent node c, therefore they can also be displayed on the same line. The parent node of text blocks d1 and d2 is different from the parent node of text blocks e1, e2, e3, and e4, therefore they are displayed on separate lines from text blocks e1, e2, e3, and e4. For example, Figure 26 The text block shown can be displayed in the second interface in the following way: Figure 26 The pasting result shown is 2.

[0477] For example, refer to Figure 27 and Figure 8 ,according to Figure 8As shown in the hierarchy, text blocks e1, e2, e3, e4, e5, and e6 share the same parent node d1, therefore they can be displayed on the same line. Text blocks f1, f2, f3, f4, f5, and f6 share the same parent node d2, therefore they can also be displayed on the same line. According to... Figure 8 It can be seen that text block d1 and text block d2 have different parent nodes, therefore text blocks d1 and d2 are displayed on different lines. Text block b belongs to a different level structure diagram than text blocks d1 and d2, therefore text block b is displayed on a separate line. For example, Figure 27 The selected text block shown can be displayed in the second interface in the following way: Figure 27 The pasting result shown is 2.

[0478] In another example, when text blocks pasted to the second interface are displayed, text blocks with the same relationship are displayed on the same line, while text blocks with different relationships or no relationship are displayed on separate lines.

[0479] When reading, people actively search for semantically related text information. For example, when looking at flight information, seeing the departure time will prompt them to immediately search for the arrival time. In other words, semantically related texts are read in a similar order. When pasting, displaying texts with the same relationship together can improve reading efficiency.

[0480] For example, refer to Figure 25 and Figure 7 Based on semantic analysis, text blocks f1, f2, f3, f4, f5, and f6 contain information related to the itinerary, specifically indicating the start and end dates. Therefore, text blocks f1, f2, f3, f4, f5, and f6 can be displayed on the same line. Text blocks g1, g2, and g3 are all seat class information; therefore, text blocks g1, g2, and g3 can also be displayed on the same line. Text block e contains price-related information, which is different from the itinerary and seat class information; therefore, it is displayed on a separate line. For example... Figure 25 The selected text block shown can be displayed in the second interface in the following way: Figure 25 The pasting result shown is 2.

[0481] For example, refer to Figure 26 Based on semantic analysis, text blocks d1 and d2 contain date-related information, indicating a subordinate relationship; therefore, text blocks d1 and d2 can be displayed on the same line. Text blocks e1, e2, e3, and e4 all contain transfer-related information; therefore, text blocks e1, e2, e3, and e4 can be displayed on the same line. Text blocks b and c belong to different entities and are therefore displayed on separate lines. For example... Figure 26The selected text block shown can be displayed in the second interface in the following way: Figure 26 The pasting result shown is 2.

[0482] For example, refer to Figure 27 and Figure 8 ,according to Figure 8 As shown in the hierarchy, text blocks e1, e2, e3, e4, e5, and e6 contain yesterday's weather information; therefore, text blocks e1, e2, e3, e4, e5, and e6 can be displayed on the same line. Text blocks f1, f2, f3, f4, f5, and f6 contain today's weather information; therefore, text blocks f1, f2, f3, f4, f5, and f6 can be displayed on the same line. Based on... Figure 8 It is known that text blocks d1 and d2 contain weather information for different dates, therefore they are displayed on separate lines. Text block b contains title information, therefore it is displayed on a separate line. For example, Figure 27 The selected text block shown can be displayed in the second interface in the following way: Figure 27 The pasting result shown is 2.

[0483] In some embodiments, when different text blocks are displayed on the same line, the text blocks can be arranged sequentially according to the sorting order of the text blocks in the tree structure (e.g., according to the order of the text blocks from top to bottom or from left to right in the first interface), for example, see reference. Figure 25 The pasting result shown is 2. Figure 26 The pasting result shown in Figure 2 and Figure 27 The pasting result is shown in Figure 2.

[0484] In other embodiments, when different text blocks are displayed in the same row, the electronic device can rearrange the order of the text blocks. For example, the text blocks can be arranged in descending order of font size. (See reference) Figure 26 As shown, the font sizes of text blocks e1, e4, e3, and e2 decrease sequentially. Therefore, when e1, e2, e3, and e4 are displayed in the same row, they are arranged in the order of e1, e4, e3, and e2. Figure 26 Paste result 4 in the middle.

[0485] When reading, people usually notice text with larger fonts, bold text, high color contrast, and unique font styles first. Sort pasted text based on these characteristics to improve reading efficiency.

[0486] In some embodiments, when different text blocks are displayed on separate lines, the text blocks can be sorted by line according to their reading order in the original layout or their hierarchy in the tree structure. For example, structures that appear earlier in the original layout will appear earlier in the second interface, such as... Figure 25 The pasting result shown is 2. Figure 26The pasting result shown is 2. Figure 27 The pasting result shown is 2. For example, the text block of the upper structure appears first in the row order of the second interface than the text block of the lower structure in the row order of the second interface. Figure 27 In the pasting result 2 shown, the "Multi-day Forecast" text block, which has a higher level, is ranked earlier than the weather information text block.

[0487] Humans typically read list-style text in an 'F' pattern, meaning that text at the top and beginning of a list is read first. This application sorts lines based on their position, improving reading efficiency.

[0488] In summary, rearranging the order and line breaks of pasted text helps restore the structural hierarchy and relationships inherent in the original layout, thus improving the efficiency and completeness of information dissemination.

[0489] In some embodiments, when the selected content is displayed on the second interface, there is only one association per row.

[0490] In some embodiments, when pasting selected text, the electronic device adds formatting marks to highlight the relationships between text and / or the hierarchical relationships between text blocks.

[0491] Adding special symbols helps to restore the structural hierarchy and relationships contained in the original layout, improving the efficiency and completeness of information dissemination.

[0492] For example, formatting marks include at least one of the following: parentheses, semicolons, dashes, serial numbers, bullets, line breaks, tabs, and vertical lines.

[0493] For example, adding line breaks can indicate parallel relationships at the same level.

[0494] For example, you can use line breaks and tabs to represent hierarchical relationships from parent nodes to child nodes. Figure 27 In the pasted result 2, the "Multi-day Forecast" and the weather information have a cross-level relationship. In some other embodiments, tabs can also be replaced with square brackets, bullet points, or serial numbers, etc.

[0495] For example, the relationships between texts include: subordinate relationship, from-to relationship (or start-to-end relationship), parallel relationship, and no relationship.

[0496] In some embodiments, a subordinate relationship can be defined between text of different levels that are close in the horizontal / vertical direction (e.g., different font size levels); or between text of different levels that are aligned in the horizontal / vertical direction (e.g., different font size levels).

[0497] For example, you can highlight the subordinate relationship between text by adding formatting marks such as parentheses and semicolons. As an example, and not a limitation, a subordinate relationship can be represented as: A(B; C). (See also...) Figure 25 Paste result 3 in the file has the following subordinate relationships: "06:10 (Shanghai Hongqiao Station)" and "07:38 (Nanjing Station)". (Reference) Figure 26 Paste result 3 in the file has the following dependency: "28 (Thursday)". (See reference) Figure 27 Paste result 3 in the middle has the following subordinate relationships: "Yesterday (07 / 02)" and "Today (07 / 03)".

[0498] In some embodiments, sibling text connected by symbols such as dashes or arrows in the original layout can be defined as having a from-to relationship.

[0499] For example, a dash can be added to highlight the "from" relationship between text. As an example, and not a limitation, a "from" relationship can be represented as: A—B (complementary text indicating the relationship). Reference Figure 25 The pasted result 3 in the image shows a 06:10 (Shanghai Hongqiao Station) — 07:38 (Nanjing Station) (1 hour 28 minutes; G1970) as the reference. Figure 27 The pasting result 3 in the middle has the following relationships: "33℃—28℃" and "33℃—27℃".

[0500] In some embodiments, text that is equally spaced on the same line can be defined as parallel, or text that is separated by the same symbol can be defined as parallel, or text that is of the same size level can be defined as parallel.

[0501] For example, adding vertical bars can highlight parallel relationships between text. As an example, and not a limitation, parallel relationships can be represented as A|B|C. (See reference) Figure 25 The pasted result 3 has the following parallel relationships: "Second Class Ticket | Second Class Ticket | Business Ticket".

[0502] For example, adding a semicolon can highlight that there is no direct relationship between the text. As an example, and not a limitation, a lack of relationship can be represented as: A; B; C. (See also...) Figure 26 Paste result 3 in the middle, the unrelated text is "Housing Provident Fund; Housing Provident Fund Management Center; Balance 615.12; ¥3.90".

[0503] In this embodiment, when pasting text, methods such as line aggregation (i.e., displaying text on the same line), line sorting (displaying text in separate lines), and adding special symbols help restore the structural hierarchy, reading order, and relationships. Therefore, when the selected text is displayed in the second interface, the text structure is clear (e.g., divided into blocks, segments, and levels), the reading order is reasonable (including reasonable reading order between structures at the same level and reasonable inclusion order between structures at different levels), and the relationships between texts are prominent. Users can read the pasted text in a consistent reading order, thereby improving the efficiency of information dissemination during text reading.

[0504] Figure 28 This illustration shows a pasting result provided by an embodiment of this application. Figure 28 As shown, when pasting one-to-N hierarchical structures with relative semantics, formatting tags can be added to reflect the correspondence between the hierarchical structures. For example, if a user selects one outbound text message and multiple return text messages, when pasting the text, bullet points, sequence numbers, or indentation can be added to reflect the many-to-one relationship between the return trip information and the outbound trip information. For instance, one outbound message can be treated as a parent node, and multiple return messages as child nodes of the outbound message.

[0505] In some embodiments, when it is detected that the text selected by the user has a header or title, even if the user does not select the corresponding header or title, it can be added to the pasted text in the form of "[Title]" when pasting.

[0506] In some embodiments, reference Figure 29 As shown in (a), when the electronic device detects that the user has adjusted the format of a text block on the second interface, it can automatically adjust the display format of text content at the same level in batches, such as... Figure 29 As shown in (b) of the diagram.

[0507] For example, electronic devices can batch adjust any display format of text content at the same level, such as font, font size, color, shadow, indentation format, and underline mark.

[0508] In some embodiments, the selected text information in this application may be pasted and displayed according to the following formatting principles:

[0509] 1) Line aggregation principle: Text blocks of the same type are displayed on the same line; text blocks belonging to the same relationship are displayed on the same line.

[0510] 2) Line break principle: Text that is not related to other text blocks in the original layout is displayed on separate lines; each line has only one related text block.

[0511] 3) Inline sorting principle: Text blocks that are at the top or left of the original layout are sorted first; upper-level text blocks are sorted before lower-level text blocks; and special symbols such as tabs can be added between upper-level and lower-level text blocks to indicate inclusion relationships.

[0512] 4) Inline sorting principles: from largest to smallest (font); from top to bottom (position); from left to right (position).

[0513] 5) The principle of highlighting the relationship between texts: Use different symbols to highlight the relationship between texts, such as the beginning and end relationship (A——B), the parallel relationship (A|B|C), and the subordinate relationship (A(B;C)).

[0514] The pasting method will be briefly described below using the first and second text blocks as examples.

[0515] In some embodiments, step S250 may specifically include: displaying the content of at least one second text block in the same row in the second interface, wherein at least one second text block satisfies a first preset condition.

[0516] The first preset condition includes at least one of the following:

[0517] At least one second text block has the same parent node;

[0518] At least one second text block has the same association relationship, which is a subordinate relationship, a parallel relationship, or a start-end relationship.

[0519] refer to Figure 7 Text blocks f1, f2, f3, f4, f5, and f6 are specific examples of at least one second text block. Alternatively, text blocks g1, g2, and g3 are each specific examples of at least one second text block.

[0520] refer to Figure 8 Text blocks f1, f2, f3, f4, f5, and f6 are specific examples of at least one second text block. Or text blocks e1, e2, e3, e4, e5, and e6 are specific examples of at least one second text block.

[0521] refer to Figure 26 Text blocks d1 and d2 are specific examples of at least one second text block. Or text blocks e1, e2, e3, and e4 are specific examples of at least one second text block.

[0522] In some embodiments, the second text block is located within the last three levels of the tree structure.

[0523] In some embodiments, when the content of at least one second text block is displayed in the same row on the second interface, the display order of the content of the at least one second text block is determined according to a seventh piece of information. This seventh piece of information includes at least one of the following: spatial information of the at least one second text block in the first interface, semantic information of the at least one second text block, or at least one of the visual salience features related to the at least one second text block in the first interface. For details regarding the seventh piece of information, please refer to the aforementioned inline sorting principles and specific embodiments of row-to-row display.

[0524] For example, the second text block that is located at the top or front of the first interface will be displayed first in the second interface, and the second text block that is located at the bottom or back of the first interface will be displayed last in the second interface.

[0525] For example, semantically related second text blocks are displayed in close order in the second interface.

[0526] For example, a second text block that has a special font, larger font size, or special text color in the first interface will be displayed earlier in the second interface.

[0527] In some embodiments, step S250 may specifically include: displaying the contents of at least one second text block and at least one fourth text block on a second interface, wherein the contents of the second text block and the contents of the fourth text block are located on different lines of the second interface, and the second text block and the fourth text block satisfy a second preset condition.

[0528] The second preset condition includes at least one of the following:

[0529] The second text block and the fourth text block have different parent nodes;

[0530] The second text block and the fourth text block have different relationships, including subordinate, parallel, or start-end relationships.

[0531] refer to Figure 7 Text blocks f1, f2, f3, f4, f5, and f6 are concrete examples of at least one second text block. Text blocks g1, g2, and g3 are concrete examples of at least one fourth text block.

[0532] refer to Figure 8 Text blocks f1, f2, f3, f4, f5, and f6 are specific examples of at least one second text block. Text blocks e1, e2, e3, e4, e5, and e6 are specific examples of at least one fourth text block.

[0533] refer to Figure 26Text blocks d1 and d2 are specific examples of at least one second text block. Text blocks e1, e2, e3, and e4 are specific examples of at least one fourth text block.

[0534] In some embodiments, the second and fourth text blocks are at least four levels below the bottom in the tree structure.

[0535] In some embodiments, when the contents of the second text block and the fourth text block are displayed in separate lines on the second interface, the display order of the contents of at least one second text block and at least one fourth text block is determined according to eighth information. This eighth information includes at least one of the following: spatial information of at least one second text block in the first interface, hierarchical information of at least one second text block, spatial information of at least one fourth text block in the first interface, or hierarchical information of at least one fourth text block. Regarding the eighth information, reference can be made to the aforementioned line sorting principles and specific embodiments of line-by-line display.

[0536] For example, if the second text block is positioned higher than the fourth text block on the first screen, then the second text block will be displayed on top of the fourth text block on the second screen.

[0537] For example, if the second text block is at a higher level than the fourth text block, then the second text block will be displayed on top of the fourth text block in the second interface, and the fourth text block will be displayed on the bottom of the second interface.

[0538] In some embodiments, method 200 may further include: displaying formatting tags on a second interface, the formatting tags being used to identify the association between at least one second text block and / or the hierarchical relationship of at least one second text block in a tree structure.

[0539] In some embodiments, method 200 may further include: detecting a user's operation to format the content of the second text block, and batch adjusting the display format of text blocks at the same level as the second text block in the second interface.

[0540] For example, the formatting operations performed by the user on the content of the second text block include at least one of the following: changing the text color, changing the text font, changing the text font size, adding / removing underlines, adding / removing serial numbers, and adjusting the indentation format.

[0541] The above text combined Figures 1 to 29 The present application describes in detail the human-computer interaction method provided in the embodiments. The following will be combined with... Figures 30 to 31 The apparatus embodiments of this application are described in detail below. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be referred to the foregoing method embodiments.

[0542] Figure 30 A schematic structural diagram of an apparatus provided in an embodiment of this application is shown. The apparatus 300 can be located in... Figure 1 The electronic device 100 shown is, or is a specific example of, the electronic device 100. The device 300 is capable of performing... Figure 6 The method shown includes each step, and can be implemented in detail. Figure 7-29 The embodiments shown are not described again to avoid redundancy.

[0543] like Figure 30 As shown, the device 300 may include a display unit 310, a recognition unit 320, and a generation unit 330.

[0544] The display unit 310 can be used to perform... Figure 6 Steps S210, S240, and S250 in method 200 are shown. The display unit 310 is used to execute display-related steps such as displaying windows, displaying application interfaces, displaying controls, and displaying pages in the aforementioned method embodiments.

[0545] The identification unit 320 can be used to perform Figure 6 Step S220 in method 200 shown. The recognition unit 320 is mainly used to perform the steps related to recognizing text content in the aforementioned method embodiments.

[0546] The generation unit 330 can be used to execute Figure 6 Step S230 in method 200 is shown. The identification unit 330 is mainly used to perform the steps related to generating the tree structure in the aforementioned method embodiments. For example, the identification unit 330 is specifically used to perform... Figure 9 Steps S231 to S233 are shown.

[0547] Optionally, the device 300 may also include a storage unit for storing program code and data of the device 300.

[0548] Figure 31 This is a schematic structural diagram of an electronic device provided in an embodiment of this application. Figure 31 The electronic device 400 shown may be Figure 1 A specific example of electronic device 100.

[0549] Figure 30 The electronic device 400 shown includes a memory 410, a processor 420, and a bus 430. The memory 410 and the processor 420 are interconnected via the bus 430.

[0550] The memory 410 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 410 may store a program, and when the program stored in the memory 410 is executed by the processor 420, the processor 420 is used to execute the various steps of the human-computer interaction method of the embodiments of this application.

[0551] The processor 420 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to perform the human-computer interaction method of the embodiments of this application.

[0552] The processor 420 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the human-computer interaction method of this application can be completed by the integrated logic circuits in the hardware of the processor 420 or by instructions in software form. The aforementioned processor 420 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 410. The processor 420 reads the information in memory 410 and executes the human-computer interaction method of the embodiments of this application in conjunction with its hardware.

[0553] In some embodiments, the electronic device 400 further includes a communication interface 440. The communication interface 440 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the electronic device 400 and other devices or communication networks.

[0554] Bus 430 may include a pathway for transmitting information between various components of electronic device 400 (e.g., memory 410, processor 420, communication interface 440).

[0555] This application also provides an electronic device, including: one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the following actions: Figure 6 The steps or implementations in the method shown are as follows: Figures 7 to 29 The specific embodiments shown are as follows.

[0556] This application also provides a readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform actions such as... Figure 6 The steps or implementations in the method shown are as follows: Figures 7 to 29 The specific embodiments shown are as follows.

[0557] This application also provides a chip that stores instructions, which, when executed by an electronic device, can realize the above-mentioned human-computer interaction method.

[0558] This application also provides a computer program product that stores a program or instructions, which can realize the above-mentioned human-computer interaction method when the program or instructions are run.

[0559] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0560] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0561] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0562] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0563] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0564] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0565] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A human-computer interaction method, characterized in that, include: Display a first interface, which includes text content; In response to the user's first action, the text content is identified; Based on the result of recognizing the text content, a tree structure of the text content is generated. The tree structure is used to represent the hierarchical information of the text content. The first node in the tree structure corresponds to a first text block, the second node associated with the first node corresponds to a second text block, and the second text block is located within the area of ​​the first text block. In response to a second user action, at least one of the second text blocks is highlighted according to the tree structure; The step of generating a tree structure of the text content based on the result of recognizing the text content includes: The text content is divided into multiple text blocks based on the interline layout similarity and the inter-column layout similarity of the text blocks. The interline layout similarity and the inter-column layout similarity of the text blocks are determined based on the spatial information of the text. The spatial information of the text includes the position of the text and the size of the text. The position of the text includes the coordinates of the text in the first interface and / or the text line index of the text in the first interface. The multiple text blocks include the first text block and the second text block. Based on the second information, the hierarchical relationship between the plurality of text blocks is obtained, wherein the second information includes at least one of the spatial information of the text, the semantic information of the text, and the visual saliency features in the first interface; Based on the third information, the text blocks in the hierarchical relationship are sorted, and the third information includes the spatial information of the text.

2. The method according to claim 1, characterized in that, The spatial information of the text is used to determine at least one of the following: the horizontal alignment of the text, the vertical alignment of the text, the text spacing, the text layout density, the horizontal alignment of the text, and the vertical alignment of the text.

3. The method according to claim 1, characterized in that, The size of the text includes both its vertical and horizontal dimensions.

4. The method according to claim 1, characterized in that, The text content is divided into multiple text blocks based on the interline layout similarity and inter-column layout similarity of the text blocks, including: The text content is segmented and the multiple text blocks are obtained based on at least one of the semantic information of the text and the visual saliency features in the first interface, as well as the interline layout similarity and inter-column layout similarity of the text blocks.

5. The method according to claim 1, characterized in that, The text content is divided into multiple text blocks based on the interline layout similarity and inter-column layout similarity of the text blocks, including: Get the first text block; At least one candidate segmentation axis is determined based on the fourth information, and the candidate segmentation axis passes through the first text block; The layout direction of the first text block is determined based on at least one of the candidate segmentation axes, wherein the layout direction of the first text block is either horizontal or vertical. The first text block is segmented based on at least one of the candidate segmentation axes to obtain multiple second text blocks, wherein the extension direction of the first segmentation axis is the same as the layout direction of the first text block.

6. The method according to claim 5, characterized in that, The fourth information includes at least one of the following: visual saliency features in the first interface, the degree of horizontal alignment of the text, the degree of vertical alignment of the text, the mapping relationship between the semantic information and spatial information of the text, and the text layout density.

7. The method according to claim 6, characterized in that, The horizontal alignment of the text is determined based on at least one of the following: differences in character size, differences in upper character boundaries, differences in lower character boundaries, and differences in the ordinates of character center points; and / or The vertical alignment of the text is determined based on at least one of the following: differences in character size, differences in left borders between characters, differences in right borders between characters, and differences in the horizontal coordinates of the center points between characters.

8. The method according to any one of claims 5 to 7, characterized in that, At least one of the candidate segmentation axes includes one or more horizontal candidate segmentation axes and one or more vertical candidate segmentation axes, and determining the layout direction of the first text block based on at least one of the candidate segmentation axes includes: The first text block is pre-segmented based on one or more of the horizontal candidate segmentation axes to obtain multiple line text regions; The interline layout similarity of the first text block is determined based on the fifth information, which includes at least one of the following: the alignment difference of the line text interval, the line width difference of the line text interval, the number of texts in the line text interval, the size difference of the text in the same column of the line text interval, and the line spacing difference of the line text interval. The first text block is pre-segmented based on one or more of the vertical candidate segmentation axes to obtain multiple column text areas; The inter-column layout similarity of the first text block is determined based on the sixth information, which includes at least one of the following: the alignment difference of the column text interval, the column width difference of the column text interval, the text number difference of the column text interval, the same-line text size difference of the column text interval, and the column spacing difference of the column text interval. The layout direction of the first text block is determined based on the inter-line layout similarity and the inter-column layout similarity, wherein... When the inter-line layout similarity is greater than the inter-column layout similarity, the first text block is laid out horizontally. When the inter-column layout similarity is greater than the inter-line layout similarity, the first text block is laid out vertically.

9. The method according to claim 8, characterized in that, Before segmenting the first text block based on a first segmentation axis among at least one of the candidate segmentation axes, the method further includes: Based on the layout direction of the first text block, the first segmentation axis is determined from at least one of the candidate segmentation axes, wherein, When the layout direction of the first text block is horizontal, the first dividing axis is the horizontal candidate dividing axis; When the first text block is laid out vertically, the first dividing axis is the candidate vertical dividing axis.

10. The method according to any one of claims 5 to 7, characterized in that, The step of determining at least one candidate segmentation axis based on the fourth information includes: Obtain the adjacency matrix of the first text block, wherein the adjacency matrix includes the vertical or horizontal distance between adjacent words in the first text block; The position in the first text block corresponding to the maximum value in the adjacency matrix is ​​determined as the candidate segmentation axis; The step of segmenting the first text block based on at least one of the candidate segmentation axes to obtain multiple second text blocks includes: Clustering the words on both sides of the first segmentation axis yields multiple second text blocks, wherein the distance between adjacent words in the second text blocks in the vertical or horizontal direction is less than the maximum value in the adjacency matrix.

11. The method according to claim 10, characterized in that, The method further includes: The position in the second text block that corresponds to the second maximum value in the adjacent matrix is ​​determined as the second dividing axis, and the second dividing axis passes through the second text block; The second text block is segmented based on the second segmentation axis to obtain multiple third text blocks, wherein the distance between adjacent words in the third text blocks in the vertical or horizontal direction is less than the second maximum value in the adjacency matrix.

12. The method according to claim 11, characterized in that, The ratio of the second-highest value to the vertical height of the smallest character within the first text block is greater than or equal to a preset threshold.

13. The method according to any one of claims 5 to 7, characterized in that, The step of obtaining the first text block includes: Obtain the horizontal alignment of each line of text and / or the vertical alignment of each column of text in the text content; The first text block is defined as a plurality of consecutive text lines and / or a plurality of consecutive text columns, wherein the horizontal alignment of the text lines is greater than a first threshold and the vertical alignment of the text columns is greater than a second threshold.

14. The method according to any one of claims 1 to 7, characterized in that, The visual salience features include at least one of the following: text color, text font, color block, shadow, and border.

15. The method according to any one of claims 1 to 7, characterized in that, The step of sorting the text blocks in the hierarchical relationship based on the third information includes: When multiple second text blocks are obtained by segmenting the first text block based on a horizontal dividing axis, the multiple second text blocks are sorted from top to bottom according to their spatial information; or... In the case where the first text block is divided into multiple second text blocks based on the vertical segmentation axis, the multiple second text blocks are sorted from left to right according to the spatial information of the multiple second text blocks; The sorting priority of the first text block is higher than that of the second text block.

16. The method according to any one of claims 1 to 7, characterized in that, Before dividing the text content and obtaining multiple text blocks based on the interline layout similarity and inter-column layout similarity of the text blocks, the method further includes: The text content is divided into multiple text lines, and each text line runs horizontally through the first interface. The multiple text lines are sorted from top to bottom to obtain the text line index of each text line. The text line index is used to divide the text content into text blocks and / or sort the text blocks.

17. The method according to any one of claims 1 to 7, characterized in that, The step of highlighting at least one of the second text blocks according to the tree structure in response to a second user action includes: Based on the trajectory of the user-selected text and the tree structure, the text block that the user intends to select is determined, wherein the trajectory of the user-selected text passes through the area of ​​at least one second text block, and the text block that the user intends to select includes at least one second text block. Highlight at least one of the second text blocks.

18. The method according to claim 17, characterized in that, The step of determining the text block that the user intends to select based on the trajectory of the text selected by the user and the tree structure includes: When the trajectory of the user-selected text is detected to switch from the region of the second text block to the region of the first text block, the hierarchy and sorting information of the second text block selected in the first text block in the tree structure are marked as a template; The system detects that the trajectory of the text selected by the user has switched from the region of the first text block to the region of the sibling node of the first node, and determines, based on the template, that the text block intended to be selected by the user includes the text block corresponding to the child node under the sibling node.

19. The method according to claim 18, characterized in that, The method further includes: Highlight the text blocks corresponding to the child nodes under the sibling node, wherein the visual salience features of the text blocks corresponding to the child nodes under the sibling node are different from the visual salience features of the second text block.

20. The method according to claim 18, characterized in that, The second operation is any of the following: selecting text based on the cursor, selecting text based on smearing, or selecting text based on clicking.

21. The method according to claim 17, characterized in that, The second operation is a text selection operation based on a click. Determining the text block the user intends to select based on the user's text selection trajectory and the tree structure includes: Based on the location clicked by the user and the boundary information of the text blocks corresponding to each node in the tree structure, determine the lowest level text block corresponding to the location clicked by the user. The lowest-level text block is determined to be the text block selected by the user's intent.

22. The method according to any one of claims 1 to 7, characterized in that, The method further includes: In response to the user's paste operation, the content of at least one of the second text blocks is displayed on the second interface according to the tree structure.

23. The method according to claim 22, characterized in that, The step of responding to the user's paste operation by displaying the content of at least one second text block on the second interface according to the tree structure includes: In the second interface, the content of at least one second text block is displayed in the same row, wherein at least one second text block satisfies the first preset condition; The first preset condition includes at least one of the following: At least one of the second text blocks has the same first node; At least one of the second text blocks has the same association relationship, which is a subordinate relationship, a parallel relationship, or a start-end relationship.

24. The method according to claim 23, characterized in that, The step of displaying at least one of the second text blocks in the same row in the second interface includes: The display order of the content of at least one second text block is determined based on the seventh information, which includes at least one of the following: spatial information of at least one second text block in the first interface, semantic information of at least one second text block, and at least one of the visual salience features in the first interface related to at least one second text block.

25. The method according to claim 22, characterized in that, The step of responding to the user's paste operation by displaying the content of at least one second text block on the second interface according to the tree structure includes: Display the contents of at least one second text block and at least one fourth text block on the second interface, wherein the contents of the second text block and the fourth text block are located on different lines of the second interface, and the second text block and the fourth text block satisfy a second preset condition; The second preset condition includes at least one of the following: The second text block and the fourth text block have different first nodes; The second text block and the fourth text block have different relationships, including subordinate, parallel, and start-end relationships.

26. The method according to claim 25, characterized in that, Displaying the contents of at least one second text block and at least one fourth text block on the second interface includes: The display order of the content of at least one second text block and the content of at least one fourth text block is determined according to the eighth information, wherein the eighth information includes at least one of the following: spatial information of at least one second text block in the first interface, hierarchical information of at least one second text block, spatial information of at least one fourth text block in the first interface, and hierarchical information of at least one fourth text block.

27. The method according to claim 22, characterized in that, The method further includes: The second interface displays format markers, which are used to identify the association between at least one of the second text blocks and / or the hierarchical relationship of at least one of the second text blocks in the tree structure.

28. The method according to claim 27, characterized in that, The formatting marks include at least one of the following: parentheses, semicolons, dashes, serial numbers, bullets, line breaks, tabs, and vertical lines.

29. The method according to any one of claims 1 to 7, characterized in that, The method further includes: The system detects that the user has adjusted the format of the content of the second text block, and then adjusts the display format of text blocks at the same level as the second text block in the second interface in batches.

30. An electronic device, characterized in that, include: One or more processors; One or more memory units; The one or more memories store one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 29.

31. A human-computer interaction device, characterized in that, Includes modules for implementing the method as described in any one of claims 1 to 29.

32. A computer-readable storage medium, characterized in that, The storage medium stores a program or instructions that, when run on a computer, cause the method as described in any one of claims 1 to 29 to be performed.

33. A chip, characterized in that, The chip stores instructions that, when executed, cause the method as described in any one of claims 1 to 29 to be performed.

34. A computer program product, characterized in that, The computer program product stores a program or instructions that, when run, cause the method as described in any one of claims 1 to 29 to be performed.

Images