Html rich text segmentation method, webpage translation method and related device

Abstract

The application discloses an HTML rich text segmentation method, a webpage translation method and related devices. The HTML rich text segmentation method comprises the following steps: obtaining a DOM tree structure corresponding to target HTML rich text; traversing nodes in the DOM tree structure in an order from top to bottom, and obtaining effective content nodes, which are element nodes of non-block container types or non-empty text nodes; taking each effective content node and a sub-tree structure thereof as a segmentation unit of the target HTML rich text, and obtaining a segmentation unit sequence formed by the segmentation units in sequence. According to the above scheme, the segmentation granularity can be effectively controlled to be the effective content nodes.

Description

Technical Field

[0001] This application relates to the field of HTML rich text processing technology, and in particular to an HTML rich text segmentation method, a web page translation method, and related apparatus. Background Technology

[0002] Web pages have corresponding HTML (HyperText Markup Language) rich text. Web pages can be news and information pages, social media application pages, etc. HTML rich text segmentation can be applied to scenarios such as web page translation (segmented unit translation), web page content moderation (segmented units are fed into a moderation model), and automatic web page summarization.

[0003] In related technologies, HTML rich text segmentation uses block-level elements as segmentation boundaries. However, this segmentation method has several limitations. Some segments are too coarse-grained, resulting in multiple semantically related paragraphs being treated as a single unit. Others are too fine-grained, causing paragraphs or even sentences with the same semantic meaning to be split into multiple segments. This lack of control over segmentation granularity makes it difficult to accurately segment content according to semantics, hindering the implementation of segmentation-based applications. Summary of the Invention

[0004] This application provides an HTML rich text segmentation method, a webpage translation method, and related apparatus, which can effectively control the segmentation granularity to effective content nodes.

[0005] This application provides a method for segmenting HTML rich text, including: obtaining the DOM tree structure corresponding to the target HTML rich text; traversing the nodes in the DOM tree structure in order from top to bottom to obtain valid content nodes, wherein the valid content nodes are element nodes of non-block-level container type or non-empty text nodes; taking each valid content node and its subtree structure as a segmentation unit of the target HTML rich text, thereby obtaining a sequence of segmentation units formed by each segmentation unit in sequence.

[0006] This application provides a webpage translation method, comprising: responding to a translation request of a webpage to be translated, obtaining the original HTML rich text and the translated HTML rich text of the webpage to be translated; segmenting the original HTML rich text and the translated HTML rich text to obtain a sequence of original text segmentation units and a sequence of translated text segmentation units; wherein the segmentation is implemented using the aforementioned HTML rich text segmentation method; based on the sequence index, obtaining the translation segmentation units associated with the original text segmentation units in the original text segmentation unit sequence from the translation segmentation unit sequence; generating the final HTML rich text based on the associated original text segmentation units and the translation segmentation units, and generating the translated webpage based on the final HTML rich text.

[0007] This application provides a segmentation device, including a conversion module, a traversal module, and a segmentation module. The conversion module is used to obtain the DOM tree structure corresponding to the target HTML rich text. The traversal module is used to traverse the nodes in the DOM tree structure in ascending order to obtain valid content nodes, which are non-block-level container element nodes or non-empty text nodes. The segmentation module is used to treat each valid content node and its subtree structure as a segmentation unit of the target HTML rich text, obtaining a sequence of segmentation units formed by these segmentation units in sequence.

[0008] This application provides a translation device, including: an acquisition module, a segmentation processing module, an association module, and a generation module. The acquisition module is used to acquire the original HTML rich text and the translated HTML rich text of the webpage to be translated in response to a translation request. The segmentation processing module is used to segment the original HTML rich text and the translated HTML rich text separately to obtain a sequence of source segmentation units and a sequence of translated segmentation units; wherein the segmentation processing is implemented using the aforementioned HTML rich text segmentation method. The association module is used to acquire, based on a sequence index, the translation segmentation units associated with the source segmentation units in the source segmentation unit sequence. The generation module is used to generate the final HTML rich text based on the associated source segmentation units and the translated segmentation units, and to generate the translated webpage based on the final HTML rich text.

[0009] This application provides a translation system, including a client and a server cluster. The client sends a translation request to the server cluster for a webpage to be translated. The server responds to the translation request by obtaining the original HTML rich text and the translated HTML rich text of the webpage to be translated. The original HTML rich text and the translated HTML rich text are segmented to obtain a sequence of source text segmentation units and a sequence of translation segmentation units. The segmentation is implemented using the aforementioned HTML rich text segmentation method. Based on the sequence index, the translation segmentation units associated with the source text segmentation units in the source text segmentation unit sequence are obtained from the translation segmentation unit sequence. The final HTML rich text is generated based on the associated source text segmentation units and the translation segmentation units, and the final HTML rich text is returned to the client. The client generates a translated webpage based on the final HTML rich text.

[0010] This application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described method.

[0011] This application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described method.

[0012] This application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0013] The above scheme obtains the DOM tree structure corresponding to the target HTML rich text, traverses the nodes in the DOM tree structure in top-to-bottom order to obtain valid content nodes, and uses the valid content nodes and their subtree structures as segmentation units. On one hand, if the HTML elements in the target HTML rich text have nested relationships, each nested HTML element will be an independent element node in the DOM tree structure. The innermost nested HTML element will be the upper-level element node, and the outermost nested element node will be the lower-level element node. Therefore, by traversing the DOM tree structure layer by layer from top to bottom, regardless of the depth of nesting of valid content nodes, the outermost non-block-level container element nodes can be ignored, penetrating to the innermost layer to obtain valid content nodes. This avoids dividing nested HTML elements into a single segmentation unit, thus preventing overly coarse segmentation granularity. On the other hand, since the valid content nodes and their subtree structures are treated as a single segmentation unit, it avoids being mistakenly divided into multiple segmentation units due to non-block-level container HTML elements embedding block-level container HTML elements, thus preventing overly fine segmentation granularity. Therefore, the segmentation granularity can be effectively controlled to be effective content nodes.

[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0016] Figure 1 This is a flowchart illustrating an embodiment of the HTML rich text segmentation method provided in this application; Figure 2A This is a schematic diagram of the HTML rich text object of this application; Figure 2B This is a schematic diagram of the DOM tree structure of this application; Figure 3 This is a flowchart illustrating an embodiment of the webpage translation method provided in this application; Figure 4 This is a schematic diagram of the structure of an embodiment of the translation system provided in this application; Figure 5 This is a flowchart illustrating a specific example of the webpage translation method described in this application; Figure 6This is a schematic diagram illustrating the original text segmentation unit sequence and the translation segmentation unit sequence obtained from the original HTML rich text in this application; Figure 7 This is an illustration of a translated webpage from this application; Figure 8 This is an illustration of the webpage to be translated in this application; Figure 9 This is another illustration of the translated webpage in this application; Figure 10 This is a schematic diagram of the structure of an embodiment of the electronic device of this application; Figure 11 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0017] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0018] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0019] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. The term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. Finally, the term "several" in this document means any integer greater than 0, such as 1, 2, 3, 4, 5, ...

[0020] In related technologies, HTML rich text is segmented by using block-level elements as segment boundaries. However, the inventors of this application have discovered that because HTML rich text allows for unlimited nesting of HTML elements, this segmentation method suffers from the following technical problems: 1. Uncontrollable segmentation granularity: Some segmentation granularity is too coarse, while others are too fine. For example, a multi-layered nested block-level container element `div` contains paragraph elements `p` and table elements. The paragraph element `p` and the table element `table` should each be independent segment units, but because this segmentation method cannot perceive the nesting level, the outermost `div` is ultimately treated as a single segment unit, resulting in overly coarse segmentation granularity. Conversely, a paragraph element `p` should be divided into one segment unit, but because it contains other block-level elements, it is ultimately divided into multiple segment units, resulting in overly fine segmentation granularity.

[0021] 2. Prone to incorrect segmentation: Some non-block-level elements are mistakenly identified as block-level elements and used as segment boundaries, leading to incorrect segmentation. For example, a p element contained in a code element may be mistakenly identified as a paragraph element, resulting in incorrect segmentation.

[0022] 3. Segmentation logic errors: Irregular nesting of HTML elements can cause segmentation logic errors. For example, HTML elements containing non-closing tags or incorrectly nested HTML elements can lead to segmentation logic errors.

[0023] To address at least some of the aforementioned technical problems, this application proposes a novel HTML rich text segmentation method. Embodiments are described below.

[0024] Figure 1 This is a flowchart illustrating an embodiment of the HTML rich text segmentation method provided in this application. Figure 1 As shown, in this embodiment, the HTML rich text segmentation method may include the following steps: S110: Obtain the DOM tree structure corresponding to the target HTML rich text.

[0025] The execution subject in this embodiment is a segmentation device, which can be any electronic device with segmentation capability.

[0026] The target HTML rich text can be the original HTML rich text of a webpage, or it can be a translated HTML rich text obtained by translating the original HTML rich text. In essence, HTML defines the basic structure of a webpage, including headings, paragraphs, lists, tables, etc.

[0027] The target HTML rich text contains multiple nested HTML elements. The HTML elements include opening and closing tags (start tag and end tag) and element content. The element content may be other HTML elements, text content containing semantics, multimedia information such as images and videos, or even whitespace values.

[0028] Each HTML element in a rich HTML text editor can be represented by a node in the DOM (Document Object Model) tree structure.

[0029] The DOM tree structure can include element nodes, text nodes, attribute nodes, etc. Specifically, the opening and closing tags of an HTML element correspond to an element node in the DOM tree, for example, " <h1>...< / h1> "These opening and closing tags are represented as an h1 element node in the DOM tree; the text content within an HTML element corresponds to a text node in the DOM tree, for example, " <h1> Hello< / h1> The text "Hello" in the HTML element is represented as a text node under the h1 element node in the DOM tree. The attributes of HTML elements correspond to attribute nodes in the DOM tree, for example, "..." In the DOM tree, the href in the element is represented as an attribute node under the element node.

[0030] Understandably, the DOM tree structure allows access to and manipulation of nested HTML elements within the target HTML rich text, each acting as an independent element node within the DOM tree. The outermost HTML element is the parent element, and the innermost element is the child element. For example, if a `div` contains a `p` element, then `div` and `p` are the parent and child elements respectively, with `p` being a child of `div`.

[0031] S120: Traverse the DOM tree structure from top to bottom to obtain valid content nodes.

[0032] Valid content nodes are element nodes that are not block-level container types or non-empty text nodes.

[0033] The first level of the DOM tree structure is the root element node at the document level. In S120, traversing from top to bottom allows us to perceive the nesting level of element nodes. In cases of multiple levels of nested element nodes, we can penetrate to the innermost layer and directly obtain the element node that carries / wrappes semantic content.

[0034] Block-level container elements include `head`, `body`, and `div`. These elements are almost always used as layout containers and do not directly carry semantic content. Non-block-level container elements include `p`, `h1`, and `table`. These elements can directly carry semantic content.

[0035] S130: Take each valid content node and its subtree structure as a segment unit of the target HTML rich text, and obtain a sequence of segment units formed by each segment unit in sequence.

[0036] After obtaining the valid content nodes, the valid content node and all its child nodes can be identified as an independent segmentation unit. After obtaining the segmentation units corresponding to all valid content nodes, the segmentation unit sequence corresponding to the target HTML rich text is obtained.

[0037] In the above embodiments, the DOM tree structure corresponding to the target HTML rich text is obtained. The nodes in the DOM tree structure are traversed from top to bottom to obtain valid content nodes. These valid content nodes and their subtree structures are then used as segmentation units. On one hand, if the HTML elements in the target HTML rich text have nested relationships, each nested HTML element will independently serve as an element node in the DOM tree structure. The innermost nested HTML element will be the upper-level element node, and the outermost nested element node will be the lower-level element node. Therefore, by traversing the DOM tree structure layer by layer from top to bottom, regardless of the depth of nesting of valid content nodes, the outermost non-block-level container element nodes can be ignored, penetrating to the innermost layer to obtain valid content nodes. This avoids dividing nested HTML elements into a single segmentation unit, thus preventing overly coarse segmentation granularity. On the other hand, since the valid content node and its subtree structure are used as a single segmentation unit, it avoids being mistakenly divided into multiple segmentation units due to non-block-level container HTML elements embedding block-level container HTML elements, thus preventing overly fine segmentation granularity. Therefore, the segmentation granularity can be effectively controlled to be effective content nodes.

[0038] On the other hand, when a valid content node is encountered during the traversal, the valid content node and its subtree structure are identified as segmentation units, so even if there is irregular nesting, it will not cause segmentation logic errors.

[0039] In some embodiments, S120 includes: determining the target level for the current traversal; for each node to be traversed in the target level, determining a first child node and a second child node among the child nodes of the node to be traversed; the first child node is a valid content node, and the second child node is an element node of block-level container type; determining the second child node as the node to be traversed in the next level; after completing the traversal of each node to be traversed in the target level, determining the next level as the target level for the current traversal, and repeating the above steps until there are no nodes to be traversed in the target level.

[0040] In the above embodiments, during the traversal of the tree structure from top to bottom, if a block-level container element node is encountered, it is treated as a node to be penetrated. When the traversal reaches its level, it is recursively entered to check if it contains valid content nodes. Thus, by using block-level container type as the sole penetration condition and non-block-level container type as segment boundaries, the traversal algorithm can locate valid content nodes regardless of the depth of nesting (e.g., 5 levels of div nesting).

[0041] To facilitate understanding, the traversal process of a tree structure will be explained below with a specific example.

[0042] like Figure 2A As shown, Figure 2A This shows an example of a target HTML rich text.

[0043] Figure 2B This is a schematic diagram of the DOM tree structure of this application. As shown in Figure 2, the DOM tree structure obtained by converting the target HTML rich text has 8 levels.

[0044] The traversal process is as follows: 1. Traverse the first level.

[0045] The first layer includes the html root node. You can start from the html root node and traverse the DOM tree structure in order from top to bottom. Specifically, the child nodes of the html root node include the head element node and the body element node. The head element node and the body element node are both block-level container type element nodes, that is, the second child nodes. The head element node and the body element node can be used as the nodes to be traversed in the second layer.

[0046] 2. Traverse the second level.

[0047] The nodes to be penetrated in the second layer include the element node head and the element node body.

[0048] The child nodes of the `head` element node are examined to determine if any valid content nodes exist within them. The child nodes of `head` include two element nodes: `meta` and `title`. Specifically: Two element nodes with the meta tag are non-block-level container type element nodes or non-empty text nodes, that is, they are effective content nodes (first child nodes). In this case, the element node meta tag can be used as a segmentation unit. It can be understood that although the element node meta tag is not used for content display, using it as a segmentation unit can effectively preserve the layout structure of HTML rich text. The element node title is treated as a valid content node (i.e., the first child node), and the element node title and its subtree structure (text) are treated as a segment unit.

[0049] The child nodes of the element node body are checked to determine whether there are valid content nodes. The child nodes of the element node body include the element node div.outer. The element node div.outer is a block-level container element node, that is, the second child node. The element node div.outer can be used as the third layer of the node to be penetrated.

[0050] 3. Traverse the third level.

[0051] The nodes to be penetrated in the third layer include the element node div.outer.

[0052] Perform feature detection on the child nodes of the element node div.outer to determine if there are any valid content nodes; the child nodes of the element node body include the element node div.layout-box. The element node div.layout-box is a block-level container type element node, that is, the second child node, and can be regarded as the fourth layer of the node to be penetrated.

[0053] 2. Traverse the 4th level.

[0054] The nodes to be penetrated in the fourth layer include the element node `div.layout-box`. The child nodes of the element node `div.layout-box` include the element node `p`, the element node `div.inner`, and the element node `table`. Among them: Element nodes `p` and `table` are non-block-level container type element nodes or non-empty text nodes, meaning they are considered valid content nodes (first child nodes). In this case, element node `p` and its subtree structure (including a text node `text"#text"First paragraph content") can be considered as a segment unit. Similarly, `table` and its subtree structure (including element nodes `thead`, `tbody`, `tr`, `th`, and `td`) can be considered as a segment unit.

[0055] The element node div.inner is a block-level container type element node, that is, the second child node. The element node div.inner can be used as the fifth layer of the node to be penetrated.

[0056] 3. Traverse the 5th level.

[0057] The nodes to be penetrated in the 5th layer include `div.inner`. The child nodes of the `div.inner` element include `span` and `p` elements. Both `span` and `p` are non-block-level container type elements or non-empty text nodes; that is, both are valid content nodes. The `span` element and its subtree structure (including a `text` element) can be treated as one segment unit, and the `p` element and its subtree structure (including a `text` element) can be treated as another segment unit.

[0058] Layers 6-8 have no nodes to be traversed, so no traversal is needed. This results in the final sequence of segmented units (segmented unit 1, segmented unit 2, segmented unit 3, segmented unit 4, segmented unit 5). Where: Segmentation Unit 1: <title> Example page< / title> " Segmentation Unit 2: The first paragraph. " Segmentation Unit 3: Part of the second paragraph. " Segmentation Unit 4: The second paragraph continues. " Segmentation Unit 5: ... " In some embodiments, S110 includes: filtering the target HTML rich text based on a preset element whitelist to obtain intermediate HTML rich text; wherein the preset element whitelist includes an HTML element whitelist, a style attribute whitelist, and a style class whitelist, the HTML element whitelist includes semantic elements, the style attribute whitelist includes display-related style attributes, and the style class whitelist includes display-related style classes; and converting the intermediate HTML rich text into a multi-level tree structure.

[0059] Specifically, after filtering the HTML elements of the target HTML rich text using an HTML element whitelist, the target HTML rich text retains only semantic HTML elements and removes script HTML elements and form HTML elements. For example, semantic HTML elements include div, p, span, h1-h6, table, ul, ol, li, blockquote, a, strong, em, etc., script HTML elements include script, style, etc., and form HTML elements include form, select, etc.

[0060] The style attribute whitelist mentioned in this application includes style attributes (CSS attributes) related to content display, such as font-size, color, and text-align. By using the style attribute whitelist, CSS attributes in the CSS stylesheet or inline style property of elements in the target HTML rich text are filtered, ensuring that the target HTML rich text retains only the CSS attributes related to content display and removes script-related CSS attributes.

[0061] The style class whitelist includes CSS class names related to content display. By filtering the class attributes of HTML elements in the target HTML rich text using the style class whitelist, the target HTML rich text retains only the CSS class names related to content display.

[0062] Understandably, filtering based on HTML element whitelists can retain semantic elements used for content display while removing non-semantic elements (such as script elements and form elements). Filtering based on style attribute whitelists (CSS attribute whitelists) can retain display-related style attributes while removing display-irrelevant style attributes (such as script-related style attributes). Filtering based on style class whitelists (class attribute whitelists) can retain display-related style classes. This achieves standardized preprocessing of the target HTML rich text, resulting in more standardized intermediate HTML rich text and avoiding interference from removed parts on segmentation.

[0063] In some embodiments, after S130, the method further includes merging semantically related and adjacent segment units in the segment unit sequence to update the segment unit sequence.

[0064] Among them, natural language models can be used to calculate the semantic content relevance of text content in adjacent segment units. If the semantic content relevance is greater than the relevance threshold, the semantic content of text content in adjacent segment units is confirmed to be related and a merging operation is performed.

[0065] In the above embodiments, merging semantically related and adjacent segmentation units can improve the semantic coherence of segmentation units.

[0066] This application also provides a webpage translation method, which is implemented based on the HTML rich text segmentation method provided in this application. An embodiment of this method is described below.

[0067] Figure 3 This is a flowchart illustrating an embodiment of the webpage translation method provided in this application. Figure 3 As shown, in this embodiment, the webpage translation method may include the following steps: S210: In response to a translation request for a webpage to be translated, obtain the original HTML rich text and the translated HTML rich text of the webpage to be translated.

[0068] Translating HTML rich text involves translating the original HTML rich text. This translation can be achieved using a translation engine, etc. Essentially, the translation process preserves the tag hierarchy and order of the original HTML rich text, translating only the text content of the text nodes to obtain the translated HTML rich text.

[0069] S220: The original HTML rich text and the translated HTML rich text are segmented to obtain the original text segmentation unit sequence and the translated text segmentation unit sequence.

[0070] The segmentation process utilizes the aforementioned HTML rich text segmentation method.

[0071] The original text segmentation unit sequence and the translation segmentation unit sequence can be, but are not limited to, array form.

[0072] S230: Based on sequence indexing, obtain the translation segmentation unit in the translation segmentation unit sequence that is associated with the original text segmentation unit in the original text segmentation unit sequence.

[0073] S240: Generate the final HTML rich text based on the associated source text segmentation units and translation segmentation units, and generate the translated webpage based on the final HTML rich text.

[0074] The above embodiments, on the one hand, involve translating the original HTML rich text into translated HTML rich text, and then segmenting the text into translation segments to obtain a sequence of translation segment units. Compared to translating the original text segment unit sequence into a translated text segment unit sequence, the translation process takes into account the context, resulting in better translation quality.

[0075] On the other hand, since both the original HTML rich text and the translated HTML rich text are segmented using the aforementioned HTML rich text segmentation method, and the HTML element hierarchy and order remain unchanged in the translated HTML rich text compared to the original HTML rich text, only the language of the text content in the text nodes is replaced, the number of original text segmentation units in the original text segmentation unit sequence and the number of translated segmentation units in the translated text segmentation unit sequence are consistent and correspond one-to-one.

[0076] Furthermore, considering the very rare discrepancies between the total number of source segmentation units in the source text segmentation unit sequence and the total number of translation segmentation units in the translation segmentation unit sequence, truncation and alignment can be performed according to the shorter sequence length. Redundant segmentation units in the longer sequence are merged into the last segmentation unit in the longer sequence. For example, if the source text segmentation unit sequence is (segmentation unit a1, segmentation unit a2, segmentation unit a3) and the translation segmentation unit sequence is (segmentation unit b1, segmentation unit b2, segmentation unit b3, segmentation unit b4, segmentation unit b5), segmentation units b4 and b5 are merged into a new segmentation unit b4, thus updating the translation segmentation unit sequence to (segmentation unit b1, segmentation unit b2, segmentation unit b3, new segmentation unit b4).

[0077] Thus, by using the same segmentation logic for full translation, original HTML rich text, and translated HTML rich text, the contradiction between "translation quality" and "segment unit alignment accuracy" is resolved. Full translation ensures better translation quality, while the same segmentation logic ensures better alignment accuracy.

[0078] In some embodiments, before S230, S310 is also included. S310: Complete the opening and closing tags of the block-level container type outside the original text segment unit, display the relevant style attributes and style classes, and update the original text segment unit.

[0079] It is understandable that if the original text segmentation unit was obtained by traversing one or more block-level container type element nodes during the traversal of the tree structure, then the opening and closing tags, display-related style attributes, and style classes (such as...) of the block-level container type element nodes will be affected. and This refers to the opening and closing tags, display-related style attributes, and style classes of the block-level container type that surrounds the original text segment unit. For example, if `p` is obtained by sequentially penetrating `div.outer`, `div.layout-box`, and `div.inner`, then the opening and closing tags, style attributes, and style classes of `div.outer`, `div.layout-box`, and `div.inner` should be added.

[0080] The above embodiments enable the original text segmentation units to maintain their original layout structure by supplementing them, so that they can be rendered and displayed on the translated webpage according to the original layout structure in subsequent processing.

[0081] In some embodiments, before S230, S320 is also included. S320: Remove HTML elements whose content is not text content from the translation segmentation unit, and modify the original display style information of the HTML elements in the translation segmentation unit to specific display style information to update the translation segmentation unit.

[0082] Specifically, for any single translation segment, HTML elements that are not used to display text content are removed from the translation segment unit. The content of these HTML elements may include multimedia information, such as images, videos, or other untranslatable elements. For example, HTML elements that are not used to display text content may include image-type HTML elements (such as the `img` element), form-type HTML elements (such as the `input` element), embedded HTML elements (such as the `embed` element), area-type HTML elements (such as the `area` element), or other HTML elements that do not need to be translated (such as the `meta` element mentioned above).

[0083] Specifically, if a source / translation segment unit contains only HTML elements that are not used to display text content, and the translation segment unit is empty after removing these HTML elements, then the translation segment unit is retained, but its content is set to null (i.e., the unit does not store any actual data) to ensure that the number of source and translation segment units is the same. Taking the meta element node as an example, if the meta element node is not an HTML element used to display text content, it may serve as a translation segment unit. In this case, the content of the translation segment unit can be cleared, and the translation segment unit is retained but its content is empty. This effectively preserves the layout structure of HTML rich text, ensuring that the number of translation and source segment units is consistent and corresponds one-to-one, without actually affecting the page rendering process or page effect.

[0084] The display style information can include display-related style attributes and style classes. Modifying the existing display style information to specific display style information aims to change the original display style / layout structure of the translation segment units to a specific display style / layout structure, such as a citation style (smaller font size, gray-orange tone). It's understandable that translation segment units can use a different layout structure than the original text segment units. For example, on a translated webpage, each translation segment unit is uniformly displayed with a citation style (smaller font size, gray-orange tone) immediately below its corresponding original text segment unit. When using a different layout structure, the translation segment units do not need to follow the layout structure of the original text segment units; therefore, the layout structure needs to be adjusted by modifying the display style information.

[0085] In some embodiments, S240 includes S330. S330: The original text segmentation units in the original HTML rich text are replaced with the original text segmentation units and the translation segmentation units associated therewith to obtain the final HTML rich text.

[0086] The replacement feature allows the original text segments and translated segments to be displayed side-by-side on the translated webpage for users to view.

[0087] In some embodiments, S340 is included after S240. S340: Store at least one of the final HTML rich text, the source text segmentation unit sequence, and the translation segmentation unit sequence into the database in response to a re-translation request for the webpage to be translated.

[0088] This involves converting the original / translated text segmentation unit sequence into JSON format and storing it in the split_content field of the database, while also recording the number of original / translated text segmentation units in the split_content_count field.

[0089] In some embodiments, the translated webpage provides a viewing mode switching button, which allows users to switch between comparison viewing mode, translation viewing mode, and original text viewing mode.

[0090] In some embodiments, the original HTML rich text, translated HTML rich text, source text segmentation unit sequence, translated text segmentation unit sequence, and final HTML rich text can be associated and stored in a database for subsequent reuse. Specifically, when the same source HTML rich text is encountered again, the associated final HTML rich text can be retrieved from the database for reuse, without needing to segment and translate it again. When encountering different source HTML rich texts but some source text segmentation units are the same, the associated source text segmentation unit sequence and translated text segmentation unit sequence can be retrieved from the database to reuse the same source text / translated text segmentation units and perform incremental translation only on different source text / translated text segmentation units.

[0091] When storing data in the database, a hash mapping can be performed on the original HTML rich text and each original text segmentation unit in the original text segmentation unit sequence to obtain a unique full hash value for the original HTML rich text and a unique segment hash value for each original text segmentation unit. The full hash value and the unique segment hash value are then associated and stored with the translated HTML rich text, the original text segmentation unit sequence, the translated segmentation unit sequence, and the final HTML rich text. The hash mapping can be implemented using hash algorithms such as MD5.

[0092] The entities performing the above-mentioned webpage translation methods can be translation systems or translation devices. Translation systems include servers / server clusters and clients, while translation devices can be servers or clients.

[0093] In some embodiments, segmentation, translation, association, webpage display, and storage are all performed by the client.

[0094] In some embodiments, the server / server cluster is responsible for translation, while the client is responsible for segmented processing, association, webpage display, and storage, thereby reducing the load on the server / server cluster.

[0095] In some embodiments, the server / server cluster is responsible for translation, segmentation, and association, while the client is responsible for webpage display and storage.

[0096] In some embodiments, the server / server cluster is responsible for translation, segmentation, association, and storage, while the client is responsible for webpage display.

[0097] This application also provides a segmentation device, including a conversion module, a traversal module, and a segmentation module. Wherein: The conversion module is used to obtain the DOM tree structure corresponding to the target HTML rich text.

[0098] The traversal module is used to traverse the nodes in the DOM tree structure in order from top to bottom to obtain valid content nodes. Valid content nodes are element nodes that are not block-level container types or non-empty text nodes.

[0099] The segmentation module is used to treat each valid content node and its subtree structure as a segment unit of the target HTML rich text, resulting in a sequence of segment units formed by these segment units.

[0100] For further detailed descriptions of the segmentation equipment, please refer to other embodiments, which will not be repeated here.

[0101] This application also provides a translation device, including an acquisition module, a segmentation processing module, an association module, and a generation module. Wherein: The acquisition module is used to respond to a translation request for a webpage to be translated, and to acquire the original HTML rich text and the translated HTML rich text of the webpage.

[0102] The segmentation module is used to segment the original HTML rich text and the translated HTML rich text to obtain the original segmentation unit sequence and the translated segmentation unit sequence; the segmentation is implemented using the aforementioned HTML rich text segmentation method.

[0103] The association module is used to retrieve, based on sequence indexing, translation segmentation units that are associated with source text segmentation units in the translation segmentation unit sequence.

[0104] The generation module is used to generate the final HTML rich text based on the associated source text segmentation units and translation segmentation units, and to generate the translated webpage based on the final HTML rich text.

[0105] For further detailed descriptions of the translation device, please refer to the preceding embodiments, which will not be repeated here.

[0106] This application also provides a translation system. Figure 4 This is a schematic diagram of the structure of an embodiment of the translation system provided in this application. Figure 4As shown, the translation system includes a client and a server cluster. The server cluster includes at least one server. The client can be a mobile terminal, desktop terminal, etc.

[0107] In some embodiments, the client sends a translation request for the webpage to be translated to the server cluster; the server responds to the translation request by obtaining the original HTML rich text and the translated HTML rich text of the webpage to be translated; the original HTML rich text and the translated HTML rich text are segmented to obtain the original text segmentation unit sequence and the translated text segmentation unit sequence; wherein, the segmentation is implemented using the aforementioned HTML rich text segmentation method; based on the sequence index, the translated text segmentation unit associated with the original text segmentation unit in the original text segmentation unit sequence is obtained in the translated text segmentation unit sequence; the final HTML rich text is generated according to the associated original text segmentation unit and the translated text segmentation unit, and the final HTML rich text is returned to the client, and the client generates the translated webpage based on the final HTML rich text.

[0108] To facilitate understanding, the webpage translation method provided in this application will be explained below with a specific example.

[0109] Figure 5 This is a flowchart illustrating a specific example of the webpage translation method described in this application. For example... Figure 5 As shown, webpage translation methods include: 1. The client sends a translation request (HTTP request) to the server cluster for the webpage to be translated.

[0110] 2. Segmented processing of server clusters.

[0111] (1) Standardized preprocessing.

[0112] The server cluster responds to translation requests by performing normalization preprocessing (HTML Purifier) ​​on the original HTML rich text of the webpage to be translated, resulting in intermediate HTML rich text.

[0113] Normalization preprocessing is implemented based on the HTMLPurifier module of the server cluster. The HTMLPurifier module is configured with whitelists for HTML elements, CSS properties of semantic elements, and CSS properties of semantic elements, and performs normalization preprocessing based on these three whitelists.

[0114] HTML element whitelist: retain semantic HTML elements (such as div, p, span, h1-h6, table, ul, ol, li, blockquote, a, strong, em, etc.), and remove script HTML elements (script, style) and form HTML elements (form, select).

[0115] A whitelist of CSS properties for semantic elements: retain display-related style properties (such as font-size, color, text-align, etc.) and filter script-related style properties.

[0116] The class attribute whitelist for semantic elements: preserves the CSS class names related to display.

[0117] (2) Obtain the original text segmentation unit sequence.

[0118] The server cluster transforms the intermediate HTML rich text (DOM tree construction / DOMDocument) to obtain the original DOM tree; and traverses the original DOM tree from top to bottom (recursive segmentation / canSplit / div penetration) to obtain the original segment unit sequence A=[a1, a2, ..., a N ], where a i This represents the i-th original text segment unit, where i = 1, 2, ..., N.

[0119] Specifically, when outputting each effective content node and its subtree structure in the original DOM tree as an original text segmentation unit, the block-level container type element tag (such as div element tag) of the outer layer of the original text segmentation unit can be supplemented, and related style attributes and style classes can be displayed to update the original text segmentation unit, thereby preserving the block-level container type element tag and maintaining the integrity of the original HTML rich text layout structure.

[0120] (3) Obtain the translation segmentation unit sequence.

[0121] The server cluster translates the intermediate HTML rich text to obtain translated HTML rich text; it then transforms the translated HTML rich text to obtain a translated DOM tree; and finally, it traverses the translated DOM tree from top to bottom to obtain the translated segment unit sequence B=[b1, b2, ..., b... N ], where b i Represents the i-th translation segment unit, b i With a i The array indices are the same.

[0122] In the process of outputting each effective content node and its subtree structure in the translation DOM tree as a translation segment unit, the element content other than the text content in the translation segment unit can be removed, such as stripping the opening and closing tags of the div element, and modifying the original display style information of the text content in the translation segment unit to the preset reference style information to update the translation segment unit.

[0123] In addition, HTML elements used to display non-text content, such as img, input, embed, and area, can be removed to ensure that the translation segment unit contains only text content. If the translation segment unit is empty after removing HTML elements used to display non-text content, the translation segment unit is retained, but its content is set to null (i.e., the unit does not store any actual data).

[0124] (4) Segmented unit association.

[0125] The server cluster associates each original text segmentation unit in the original text segmentation unit sequence with the corresponding translation segmentation unit at the same array index in the translation segmentation unit sequence. Where, b i With a i Related.

[0126] The acquisition of the original text / translation segmentation unit sequence and the association of segmentation units are achieved through the segmentation processing module of the server cluster.

[0127] (5) Database persistence.

[0128] After converting the original / translated segmentation unit sequence into JSON format, the server cluster stores it in the split_content field and records the number of segmentation units in the split_content_count field.

[0129] (6) The translated webpage is displayed.

[0130] When a user triggers page translation, the client can obtain the split_content field corresponding to the page from the server cluster, which is the sequence of source text segmentation units and the sequence of translation segmentation units. Based on the associated source text segmentation units and translation segmentation units, the client generates the final HTML rich text and renders the final HTML rich text as the translation page.

[0131] Specifically, upon receiving the `split_content` field, the client can parse it to obtain the source text segmentation unit sequence and the translation segmentation unit sequence. Then, a main container is created in the corresponding DOM tree of the page to hold all the source text segmentation units and translation segmentation units. The source text segmentation unit sequence and the translation segmentation unit sequence are traversed, and corresponding HTML elements are created for each group of source text segmentation units and their associated translation segmentation units to obtain the final HTML rich text for page rendering.

[0132] More specifically, for any original text segment unit, a div element can be created based on the original text segment unit, and its class name is initialized to text-original. The text-original includes the original display-related style attributes and style classes of the original text segment unit. At the same time, for any translation segment unit, a div element can be created based on the translation segment unit, and its class name is initialized to text-translated. The text-translated includes the original preset reference style information of the translation segment unit. Then, the div elements corresponding to the original text segment unit and the div elements corresponding to the translation segment unit are loaded into the main container in sequence. The div element corresponding to the translation segment unit can be used as an HTML element nested within the div element corresponding to the original text segment unit.

[0133] The client displays the translated webpage corresponding to the final HTML rich text. The original text segment units are rendered using the normal style "original layout structure", while the translated segment units are rendered using the preset reference style, so as to display the original text segment units and the translated segment units in contrast.

[0134] Figure 6 This is a schematic diagram illustrating the original text segmentation unit sequence and the translation segmentation unit sequence obtained from the original HTML rich text in this application. For example... Figure 6 As shown, the original HTML rich text is fully translated to obtain the translated HTML rich text. Segmentation processing is performed on the original HTML rich text to obtain the original text segmentation unit sequence A = [a1, a2, a3, a4], and segmentation processing is performed on the translated HTML rich text to obtain the translation segmentation unit sequence B = [b1, b2, b3, b4]. Specifically, block-level container type tags are retained in the original text segmentation unit sequence A, such as adding div tags to a1. For the translation segmentation unit sequence B, element tags other than text content are removed from the translation segmentation units, such as stripping the opening and closing tags of div elements, and HTML elements not used to display text content are also removed, such as the HTML element in b3. The element tag and its content, i.e., the image has been removed.

[0135] Figure 7 This is an illustration of a translated webpage from this application. (For example...) Figure 7 As shown, the original text segments in A and their corresponding translation segments in B are displayed on the translated webpage in the client.

[0136] Furthermore, a translation button can be provided on the webpage, allowing users to switch between viewing modes (comparison viewing mode, translation viewing mode, and original text viewing mode) by clicking the translation button.

[0137] Figure 8This is an illustration of the webpage to be translated in this application. Figure 9 This is another illustration of the translated webpage in this application. (For example...) Figure 8-9 As shown, the webpage to be translated contains an English title and three paragraphs. The translated webpage is obtained by translating the webpage from English to Chinese. In the translated webpage, the translated Chinese title follows the English title in a quotation style. Each translated Chinese paragraph follows its corresponding English paragraph in a quotation style.

[0138] Figure 10 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Figure 10 As shown, the electronic device 50 includes a memory 51 and a processor 52. The processor 52 is used to execute program instructions stored in the memory 51 to implement the steps in any of the above method embodiments. In a specific implementation scenario, the electronic device 50 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 50 may also include a laptop computer, a tablet computer, or other carrier device, which is not limited here.

[0139] Specifically, processor 52 controls itself and memory 51 to implement the steps in any of the above method embodiments. Processor 52 may also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 52 may be implemented using integrated circuit chips.

[0140] Please see Figure 11 , Figure 11 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 60 stores program instructions 601 thereon, which, when executed by a processor, implement the steps in any of the above method embodiments.

[0141] This application also provides a computer program product comprising a computer program that, when executed by a processor, can implement the steps of the methods described in any of the foregoing embodiments. Specifically, the computer program product can be a software or program product containing a computer program, capable of running on a computing device or stored on any available medium.

[0142] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0143] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. In another image location, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0145] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for segmenting HTML rich text, characterized in that, include: Obtain the DOM tree structure corresponding to the target HTML rich text; Traverse the nodes in the DOM tree structure from top to bottom to obtain valid content nodes, where valid content nodes are element nodes that are not block-level container types or non-empty text nodes. Each effective content node and its subtree structure are taken as a segment unit of the target HTML rich text, resulting in a sequence of segment units formed by the segment units in sequence.

2. The method according to claim 1, characterized in that, The step of traversing the nodes in the DOM tree structure from top to bottom to obtain valid content nodes includes: Determine the target level for the current traversal; For each node to be penetrated in the target hierarchy, a first child node and a second child node are determined among the child nodes of the node to be penetrated; the first child node is the valid content node, and the second child node is an element node of the block-level container type; The second child node is identified as the next level node to be penetrated; After traversing each node to be penetrated in the target level, the next level is determined as the current target level for traversal, and the above steps are repeated until there are no nodes to be penetrated in the target level.

3. The method according to claim 1, characterized in that, The process of obtaining the DOM tree structure corresponding to the target HTML rich text includes: The target HTML rich text is filtered based on a preset element whitelist to obtain intermediate HTML rich text; wherein, the preset element whitelist includes an HTML element whitelist, a style attribute whitelist, and a style class whitelist, the HTML element whitelist includes semantic elements, the style attribute whitelist includes display-related style attributes, and the style class whitelist includes display-related style classes. The intermediate HTML rich text is converted into the DOM tree structure.

4. The method according to claim 1, characterized in that, After taking each effective content node and its subtree structure as a segment unit of the target HTML rich text, and obtaining a sequence of segment units formed by the segment units in sequence, the method further includes: The segmentation units that are related and adjacent in the text content of the segmentation unit sequence are merged to update the segmentation unit sequence.

5. A webpage translation method, characterized in that, include: In response to a translation request for a webpage to be translated, obtain the original HTML rich text and the translated HTML rich text of the webpage to be translated; The original HTML rich text and the translated HTML rich text are segmented to obtain a sequence of original segmentation units and a sequence of translated segmentation units; wherein the segmentation is implemented using the method described in any one of claims 1-4; Based on sequence indexing, the translation segmentation unit associated with the original text segmentation unit in the original text segmentation unit sequence is obtained in the translation segmentation unit sequence; The final HTML rich text is generated based on the associated source text segmentation units and translation segmentation units, and a translated webpage is generated based on the final HTML rich text.

6. The method according to claim 5, characterized in that, Before obtaining the translation segmentation unit associated with the original text segmentation unit in the translation segmentation unit sequence based on sequence indexing, the method further includes: Complete the opening and closing tags of the block-level container type outside the original text segmentation unit, display related style attributes and style classes, and update the original text segmentation unit.

7. The method according to claim 5, characterized in that, Before obtaining the translation segmentation unit associated with the original text segmentation unit in the translation segmentation unit sequence based on sequence indexing, the method further includes: Remove non-text HTML elements from the translation segmentation unit and modify the original display style information of the HTML elements in the translation segmentation unit to specific display style information to update the translation segmentation unit.

8. The method according to claim 5, characterized in that, The step of generating the final HTML rich text based on the associated source text segmentation units and translation segmentation units includes: The original text segmentation unit and its associated translation segmentation unit are used to replace the original text segmentation unit in the original HTML rich text to obtain the final HTML rich text.

9. The method according to claim 5, characterized in that, After generating the final HTML rich text based on the associated source text segmentation units and translation segmentation units, the process includes: The final HTML rich text is stored in the database in response to a re-translation request for the webpage to be translated.

10. A segmentation device, characterized in that, include: The conversion module is used to obtain the DOM tree structure corresponding to the target HTML rich text; The traversal module is used to traverse the nodes in the DOM tree structure in order from top to bottom to obtain valid content nodes, where the valid content nodes are element nodes that are not block-level container types or non-empty text nodes. The segmentation module is used to take each of the effective content nodes and its subtree structure as a segmentation unit of the target HTML rich text, so as to obtain a segmentation unit sequence composed of the segmentation units in sequence.

11. A translation device, characterized in that, include: The acquisition module is used to respond to a translation request for a webpage to be translated, and to acquire the original HTML rich text and the translated HTML rich text of the webpage to be translated. The segmentation processing module is used to segment the original HTML rich text and the translated HTML rich text respectively to obtain the original text segmentation unit sequence and the translated text segmentation unit sequence; wherein, the segmentation processing is implemented using the method described in any one of claims 1-4; The association module is used to obtain, based on sequence indexing, translation segmentation units that are associated with original text segmentation units in the original text segmentation unit sequence within the translation segmentation unit sequence; The generation module is used to generate the final HTML rich text based on the associated source text segmentation units and translation segmentation units, and to generate the translated webpage based on the final HTML rich text.

12. A translation system, characterized in that, It includes a client and a server cluster, wherein the client is used to send a translation request for the webpage to be translated to the server cluster; The server is used to respond to the translation request and obtain the original HTML rich text and the translated HTML rich text of the webpage to be translated; The original HTML rich text and the translated HTML rich text are segmented to obtain the original text segmentation unit sequence and the translation segmentation unit sequence, respectively; wherein, the segmentation is implemented using the method of any one of claims 1-4; based on the sequence index, the translation segmentation unit associated with the original text segmentation unit in the original text segmentation unit sequence is obtained in the translation segmentation unit sequence; The final HTML rich text is generated based on the associated source text segmentation units and translation segmentation units, and the final HTML rich text is returned to the client, which then generates a translated webpage based on the final HTML rich text.

13. An electronic device, characterized in that, It includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method of any one of claims 1 to 9.

14. A computer-readable storage medium / program product, characterized in that, It stores program instructions / computer programs that, when executed by a processor, implement the method of any one of claims 1 to 9.

Images