A front-end dynamic screenshot translation method, system, device and storage medium

Abstract

The application discloses a front-end dynamic screenshot translation method, system, device and storage medium. The application uses a method for recognizing screenshot text for translation, so that a user can freely select any area on a page for translation, breaking the limitation of the translation area and meeting diversified translation requirements. Text positioning based on coordinate mapping is adopted, a text area is accurately matched through calculation of a matching threshold, in a translation rendering stage, a node cloning and text replacement technical means is adopted, and adjustment rules for CSS layout are carried out, so that the problem of page structure confusion caused by non-matching of multilingual text length is solved, translation display is more stable and standard, and it is ensured that the user browsing experience is not affected, a mixed layout rendering method is adopted, the needs of a user for simultaneously viewing original text and translation are met, and user translation experience is improved. The translation of the application is only completed in the front end, decoupling with a business end is realized, translation storage cost and maintenance cost are reduced, and front-end page translation real-time response is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, more particularly, to a front-end dynamic screenshot translation method, system, device and storage medium. BACKGROUND

[0002] Under the driving of global scientific research collaboration and knowledge sharing, data opening needs to gradually evolve towards internationalization. For example, data hub platforms need to aggregate data resources from different disciplines such as astronomy, biology, and geology, and integrate multi-model analysis capabilities. Therefore, the international version needs to provide multilingual support capabilities to users. Traditional internationalization solutions mainly rely on two paths: (1) Text pre-translation: generating multiple language versions during data storage. This approach leads to serious data redundancy and is difficult to update in real time. When a Chinese document is revised, the translation process for all languages needs to be triggered again, which significantly delays the process. (2) Global non-discriminatory translation: Using Google Translate plug-ins, although real-time translation can be achieved, the flexibility of translation is lost. Non-discriminatory translation in this way can lead to distortion of non-text elements (incorrectly identifying formulas and pictures as natural language) and inaccurate professional terms.

[0003] There are several solutions to front-end page translation at present:

[0004] The page translation method disclosed in the patent with publication number CN112395890A creates multiple language type files to store translated texts of different language types, and separately packages each language type file as a language package. Before users access the webpage, the developer needs to organize and set the displayed texts, complete the language package construction, and then load the corresponding language translation texts according to the user's language type on demand to achieve webpage translation. However, this method has a huge construction and maintenance cost in the early stage. If the translation requirements change or the page content adjusts, the workload of maintaining the language package will increase dramatically. In addition, as the number of pages and translation content increases, the database space occupied by the language package storage will continue to expand, causing resource waste.

[0005] The page translation method disclosed in the patent with publication number CN115630615A detects the user operation triggered by the target user in the page, obtains the target page information corresponding to the user operation, then translates the collected page information and replaces the original text to achieve the purpose of page translation. However, this method essentially belongs to non-discriminatory page translation. That is, when the user operation is obtained, the entire translated text is translated, so there is still a problem of identifying formulas and pictures as natural language. At the same time, the flexibility of translation is lost. The user cannot specify the area to be translated after triggering the translation operation.

[0006] The patent with the publication number CN113392660A also realizes page translation through screenshot mode, but since the translation is displayed in a floating layer or a specific area in the target page, it will cause dynamic layout interference. When the webpage contains responsive layout, floating elements or other dynamic loading content, the floating layer (original text translation toolbar) will destroy the original structure and layout of the page, causing the page to display incorrectly and affecting the user's browsing and interaction experience of the overall content of the page. SUMMARY

[0007] The present application aims to solve the problems of the prior art and provides a front-end dynamic screenshot translation method, system, device and storage medium. The present application solves the following technical problems:

[0008] 1. Language output real-time. Different sources of data resources have language mixing problems. For example, the data set provided by Deepmind is introduced in English, and the model introduction language provided by the National Astronomical Observatory is Chinese. The switching of language needs to rely on the language of the data publisher uploaded resources, so the development of the international version of the platform usually needs to rely on manual maintenance of multiple language versions. The present application dynamically identifies and translates the specified block in the front end, so that the translation is only completed in the front end, without pre-translation and manual intervention. The data publisher only needs to output a single language, and the front-end dynamic translation result reduces the pressure of multi-language adaptation when publishing resources.

[0009] 2. Precise translation of terminology protection. Different fields of professional terms have professional language correspondence, and direct translation using a translator will cause misinterpretation, making it difficult for international users to understand. The present application binds related terminology library in the front-end area that needs to be translated, such as astronomical terminology library. When the user switches the language, "accretion disk" is automatically matched to the terminology library and translated into "accretion disk".

[0010] 3. Mixed interaction of international version UI. When a user views data details, he / she often needs to translate part of the document content, and these "parts" are not complete hierarchical structures in the page and may be distributed across levels or discretely. The present application uses screenshot translation mode to flexibly determine the translation area. At the same time, CSS+DOM operation is used in the translation rendering stage to support mixed layout and meet the user's need to view the original text and the translation at the same time.

[0011] According to a first aspect of the present specification, a front-end dynamic screenshot translation method is provided, which comprises:

[0012] Marking a label for defining the translation boundary in the front-end project;

[0013] Listening to the front-end page screenshot event, when the user triggers the screenshot and uploads the screenshot snapshot, the following operations are performed in sequence: extracting the text in the screenshot, target language translation, and translation rendering;

[0014] The text in the extracted screenshot includes: converting image coordinates into page coordinates to locate the position of the text in the screenshot on the page; obtaining a DOM element stack corresponding to the position, traversing the DOM element stack in combination with the hierarchical attributes of the page, reading out the stack elements to obtain candidate text nodes, performing screening on the text nodes at the edge of the screenshot based on a matching threshold, and finally obtaining the text nodes to be translated;

[0015] The target language translation includes: replacing the matched proper nouns based on a local terminology library, and performing sorting based on a word length descending order algorithm to preferentially match long words and replace them;

[0016] The translation rendering includes: taking the text nodes to be translated as source nodes and cloning them to generate translation copy nodes that are identical in structure and inline style, inserting the copy nodes into the DOM tree as sibling adjacent nodes of the source nodes, adjusting the CSS style of the copy nodes to complete translation container modification, and realizing instant switching between only translation and bilingual comparison rendering modes by manipulating the CSS display attribute state of the source nodes and the copy nodes.

[0017] Further, the label for limiting the translation boundary includes <translate>Labels and <no-translate>Labels, said <no-translate>The text in the area of the label wrap does not need to be translated, said <translate>The label adds custom data attributes for setting attribute limits for proper nouns.

[0018] Further, the coordinate conversion formula for converting image coordinates to page coordinates is as follows: (viewportX, viewportY) = (imgX+clientX-scrollX, imgY+clientY-scrollY), where (viewportX, viewportY) is the positioning of the recognized text on the page, (imgX, imgY) is the image local coordinate of the text block detected by the text recognition technology, (clientX, clientY) is the starting position of the user's screenshot mouse press, and (scrollX, scrollY) is the position of the page scroll.

[0019] Further, after traversing the DOM element stack and reading out the stack elements, one or both of the following optimization processes are performed to obtain the candidate text nodes:

[0020] a. Traverse the DOM nodes and filter out the DOM nodes that do not need to be translated as pre-annotated by the label;

[0021] b. Use the document.createTreeWalker() method to create a TreeWalker and specify the node type to be traversed as text nodes; use depth-first node traversal, and if the next node does not exist, stop recursion; return all matched text nodes.

[0022] Further, the text nodes on the edge of the screenshot are screened based on a matching threshold, specifically: the overlapping area of the text region corresponding to the text node on the edge of the screenshot and the screenshot region is calculated; a matching threshold is pre-set, if the overlapping area accounts for more than the matching threshold percentage of the text region area, it is considered that the node needs to be translated, otherwise it is considered that the node does not need to be translated.

[0023] Further, the local terminology library is associated with the data attribute configured by the label for replacing proper nouns; the priority matching of long words includes matching original long words and matching translated long words, and the replacement of proper nouns includes: converting the proper nouns into a regular pattern with boundaries, matching all proper nouns according to the local terminology library and performing boundary processing, and using the replace method provided by the regular API to perform text replacement.

[0024] Further, based on the character length expansion rate of the translation relative to the original text, the copy node CSS style is adjusted based on an elastic adjustment strategy, specifically:

[0025] Primary strategy: when the expansion rate <= 100%, use the CSS Flexbox model feature to realize the elastic allocation of container space by configuring container scaling flex-grow / shrink;

[0026] Secondary strategy: when the expansion rate > 100% and <= 200%, calculate the non-linear scaling rate based on the expansion rate using the square root function, and scale the font size according to the calculated scaling rate;

[0027] Tertiary strategy: when the expansion rate > 200%, trigger automatic line wrapping and vertical scroll bar injection.

[0028] According to a second aspect of the present specification, a front-end dynamic screenshot translation system is provided, which is used to implement the front-end dynamic screenshot translation method as described in the first aspect, and the system comprises:

[0029] Label setting module: marking labels for defining translation boundaries in the front-end project;

[0030] Text extraction module: extracting text within the screenshot after listening to user triggering of screenshot and uploading of screenshot snapshots, comprising: converting image coordinates to page coordinates to locate the position of text within the screenshot on the page, obtaining a DOM element stack corresponding to the position, traversing the DOM element stack in combination with the hierarchical attributes of the page, reading out the stack elements to obtain candidate text nodes, performing matching threshold-based screening on text nodes at the edges of the screenshot, and finally obtaining text nodes to be translated;

[0031] Target language translation module: replacing matched proper nouns based on a local terminology library, preferentially matching long words and replacing them based on a word length descending order sorting algorithm, and calling a server translation interface to translate the text after replacing the proper nouns;

[0032] Translation rendering module: cloning the text nodes to be translated as source nodes to generate translation copy nodes that are identical in structure and inline style, inserting the copy nodes into the DOM tree as sibling nodes adjacent to the source nodes, adjusting the CSS style of the copy nodes to complete translation container modification, and realizing instant switching between only translation and bilingual comparison rendering modes by manipulating the CSS display attribute state of the source nodes and the copy nodes.

[0033] According to a third aspect of the present specification, an electronic device is provided, comprising a memory and a processor, the memory being coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to realize the front-end dynamic screenshot translation method as described in the first aspect.

[0034] According to a fourth aspect of the present specification, a computer readable storage medium is provided, which stores a computer program, the program being executed by a processor to implement the front-end dynamic screenshot translation method according to the first aspect.

[0035] According to a fifth aspect of the present specification, a computer program product is provided, which includes computer programs / instructions, the computer programs / instructions being executed by a processor to implement the front-end dynamic screenshot translation method according to the first aspect.

[0036] The beneficial effects of the present application are:

[0037] 1. Breaking the limitation of the translation area: the present application uses the method of identifying screenshot text for translation, allowing users to freely define any area on the page for translation, breaking the limitation of the translation area and meeting the diversified translation needs.

[0038] 2. Optimizing the display effect of the translation: in view of the problem that the floating layer display of the translation may cause the disorder of the document structure, the present application uses the technical means of node cloning and text replacement to process the translation of the screenshot content, so that the translation display is more stable and standardized, avoids interference with the original structure of the document, and ensures that the user browsing experience is not affected.

[0039] 3. Improving translation efficiency and reducing storage cost: the present application realizes real-time response of front-end page translation by defining the translation window through the innovative way of screenshot, and the user operation feedback is instant. Moreover, the present application does not need to pre-store translation files, which effectively reduces the database storage by more than 60% compared with the traditional mode. This pure front-end translation mode also realizes decoupling with the back-end, reduces the interaction cost between the front-end and the back-end, and improves the overall performance of the system. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described in the following only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0041] Figure 1 The overall flowchart of the front-end dynamic screenshot translation method shown for an exemplary embodiment;

[0042] Figure 2 The coordinate conversion schematic diagram shown for an exemplary embodiment;

[0043] Figure 3 The page space conversion schematic diagram shown for an exemplary embodiment;

[0044] Figure 4 A diagram of a DOM element stack structure shown for an exemplary embodiment;

[0045] Figure 5 An enlarged detail view of an overlapping portion of an edge node text region and a screenshot region shown for an exemplary embodiment;

[0046] Figure 6 A diagram of a structure of an electronic device shown for an exemplary embodiment. DETAILED DESCRIPTION

[0047] For better understanding of the technical solutions of the present application, the embodiments of the present application are described in detail below with reference to the drawings.

[0048] It should be clear that the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0049] The terms used in the embodiments of the present application are only for the purpose of describing the specific embodiments, and are not intended to limit the present application. The singular forms "a", "an" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0050] The implementation of the front-end dynamic screenshot translation method provided by the present application includes two parts, which are manual intervention and machine intervention.

[0051] I. Manual intervention

[0052] Manual intervention refers to manually marking <translate> <no-translate>Tags, in a declarative syntax, define translation boundaries. Specifically:

[0053] 1. Tag <no-translate>Region. With <no-translate>The text in the wrapped area will not be translated, such as formulas, charts, codes, etc. For example, in a document for building a model, the user is usually shown operation code, and the code part does not need to be translated. Therefore, the code part can be wrapped with a <no-translate>< / no-translate> label to avoid being recognized as natural language by the machine.

[0054] 2. Limit the translation boundary according to the semantic context, so that <translate>The tag adds data-domain attribute. The astronomical data set has a special astronomical vocabulary in translation, in order to avoid the machine mis-translation of proper nouns and set attribute limits. For example, <translate data-domain="astronomy">Accreting disks are key structures for star formation< / translate> .

[0055] By design <translate>and <no-translate>Tag, accurate division of translation boundaries, avoid machine errors to identify formulas, charts, code and other information as natural language.

[0056] II. Machine intervention

[0057] In this phase, the front-end page screenshot event is continuously monitored. When the user triggers the screenshot and uploads the screenshot snapshot, the following operations are performed in turn: extracting the text in the screenshot, target language translation, and translation rendering, as shown in Figure 1 The specific implementation process of the machine intervention phase is described in detail below.

[0058] 1. The user triggers the screenshot and uploads the screenshot snapshot. The screen stream is obtained through the browser API, and the stream media is converted into a binary file and uploaded to the server.

[0059] 2. Extracting text in the screenshot. This step mainly refers to the synchronous mapping of in-image text and Document Object Model (DOM) nodes. The in-image text recognition method includes but is not limited to Optical Character Recognition (OCR) technology. The specific steps of extracting the text in the screenshot are as follows:

[0060] (1) Coordinate conversion. Convert the local coordinates of the image to the global coordinates of the page. Since both the image and the page can be represented by pixels, the image coordinates and the page coordinates can be converted, thereby locating the position of the text in the screenshot to the position of the text on the page. The coordinate conversion formula is as follows:

[0061] (viewportX,viewportY) = (imgX+clientX-scrollX,imgY+clientY-scrollY)

[0062] where (viewportX,viewportY) is the location of the recognized text on the page, (imgX,imgY) is the local image coordinates of the text block detected by the OCR technology, (clientX,clientY) is the starting position of the user's screenshot mouse press, and (scrollX,scrollY) is the position of the page scroll. As shown in Figure 2 the dashed rectangular box is the text to be translated, and there is no current position scrolling, so the coordinates of the text relative to the page are (20+400-0,127+214-0) = (420,341).

[0063] (2) Text node positioning. Since the coordinates obtained in step (1) are two-dimensional, while the actual front-end page structure is three-dimensional, when the page has a floating layer such as a pop-up window, a dialog box, a drawer, etc., the picture taken by the user may have a text overlap situation, therefore, the text positioned by using the coordinates obtained in step (1) includes both the displayed text of the page and the overlapped text. Taking the page space conversion schematic diagram as an example, it is assumed that the user screenshot is only the pop-up window part on the left, and the text positioned by the coordinates includes both "Do you want to download this dataset?" and "For a 7 billion parameter large model..." two parts. The text node positioning is specifically as follows: Figure 3 Figure 3

[0064] ① Based on the browser API: document.elementsFromPoint(viewportX, viewportY) to obtain the DOM element stack corresponding to the coordinate position on the page. This API can store the three-dimensional page structure in a linear manner. As shown in the following figure, the DOM element stack obtained by using this API is stored in memory according to "page level", "from bottom to top". Specifically, the stack element stored at the top of the stack is the top matching DOM node of the page, and the stack element stored at the bottom of the stack is the bottom matching DOM node of the page, and the matching DOM node is stored from the child node to the parent node to the root node in this way "from bottom to top". Figure 4

[0065] ② Traverse the DOM element stack. Due to the first-in-first-out nature of the stack, the top element of the stack must contain the text node to be matched. In addition, assuming that the page has a text overlap situation, the DOM element stack needs to be continuously traversed until the next layer of the page structure is reached and the top node in the stack structure of the next layer. The page level is distinguished according to the Z-index attribute, and the front-end developer sets the Z-index attribute for a node, and all child nodes from the node will be layered displayed on the page. Therefore, the Z-index attribute can be used as an identifier for distinguishing the page level. If the node is set with the Z-index attribute, the next node of the node must be the next level. Specifically: first, the top element of the stack is popped; second, traverse to the next stack element, if the current traversal element is the root node of the current page level, that is, the Z-index attribute exists, it is considered that the current level traversal is completed, and the next element to be traversed is popped.

[0066] ③ Read the popped element. Based on the DOM node obtained in step ②, the target text node to be translated is obtained. These DOM nodes can be optimized in the following two ways:

[0067] ​​​a. Filter DOM nodes that do not need translation. Traverse the DOM nodes and filter out those that use <no-translate>wrapped DOM node. The above-mentioned text need not be translated when <no-translate>Label wrapping.

[0068] b. The Html document structure is a multi-level tree structure, due to the offset of the coordinate distance from the text (the starting coordinate is not necessarily the first pixel of the first character of the text to be translated, which is considered to be within a reasonable range), the DOM node located by the same coordinate has two cases, one is directly located to the text node to be translated, and the other is located to a certain upper parent node of the text node to be translated. Therefore, this step needs to further extract the text of the stack elements traversed. Specifically: use the document.createTreeWalker() method to create a TreeWalker, and specify the node type to be traversed as NodeFilter.SHOW_TEXT (text node); use depth-first node traversal, if the next node does not exist, stop recursion; return all matched text nodes.

[0069] The above two processing methods are non-essential steps, and one or both can be selected for processing according to the specific situation.

[0070] ④ Accurate matching of the text area. Since the user may not completely capture the edge text when actually capturing the screenshot, this step accurately matches the text area by calculating the matching threshold to filter out the edge text that does not meet the conditions and improve the translation accuracy. Specifically:

[0071] a. Calculate the overlapping area of the text area corresponding to the text node located at the edge of the screenshot and the screenshot area;

[0072] b. Calculate the coincidence rate. A matching threshold is set in advance, and in this embodiment, the threshold is set to 80%, that is, when the overlapping area is more than 80% of the edge node text area, it is considered that the edge node needs to be translated, otherwise the translation of the edge node is discarded;

[0073] For example, as shown in Figure 5 , assuming that the large rectangle is the user's screenshot area and the small rectangle is the text area corresponding to the text node located at the edge of the screenshot, the text in the small rectangle is not completely in the screenshot area. Label the top left corner of the rectangular box as the starting point. Assuming that the starting point of the text area is A and the starting point of the screenshot area is B, the starting point of the overlapping part is C, as shown in Figure 5 . Among them, the length of the text area is 105 and the width is 32, so the overlapping area is: (105-(84-72))*32=2976, accounting for 88.6% (2976 / 3360*100%) of the text area area 105*32=3360, and the overlapping area is more than 80% of the edge node text area. It is considered that the edge node needs to be translated.

[0074] 3. Target language translation. This step is a further operation based on the parsed text nodes to be translated in the above steps.

[0075] (1) Proper name replacement. Replace the matched proper name based on the local terminology library. Where the local terminology library is pre-constructed to store the proper names of different disciplines. The terminology library is associated with the source language and the target language. The proper name replacement is implemented by the following steps: <translate>The data-domain attribute configured in the tag. For example: "POAG": {"zh-CN": "Primary open-angle glaucoma", "ja": "Primary open-angle green cataract"}.

[0076] (2) A sorting algorithm based on descending word length prioritizes matching and replacing longer terms. Prioritizing long term matching includes matching both long terms in the original text and long terms in the translation. For example, both open-angle glaucoma and glaucoma correspond to "POAG" in English. Therefore, the noun replacement table needs to store two key-value pairs: "glaucoma": {"en": ["POAG"], "ja": "green cataract"}, and "open-angle glaucoma": {"en": ["POAG"], "ja": "green cataract"}. In this case, prioritizing short term matching would lead to inaccurate translations, such as "open-angle glaucoma" -> "open-angle POAG" -> "Open-angle POAG", resulting in reduced translation accuracy. By prioritizing the matching of the most complete long terms, the algorithm avoids the inability to recognize long terms due to prioritizing short terms, further ensuring semantic integrity. Due to the limited number of Chinese characters and the abundance of foreign characters, for example: open-angle glaucoma -> POAG / Primary Open-Angle Glaucoma, longer terms usually correspond to a greater expansion rate. Prioritizing the matching of longer words can provide early warning of layout risks.

[0077] Specifically: First, proper nouns are converted into bounded regular expression patterns, where the Chinese bounded regular expression is: [\u4e00-\u9fa5]. For example, given the input text "open-angle glaucoma detection", according to the pre-built terminology database, the proper nouns "open-angle glaucoma" and "glaucoma" are matched. Then, the bounded regular expressions are processed to obtain: {pattern: / (^|[^\u4e00-\u9fa5])open-angle glaucoma([^\u4e00-\u9fa5]|$) / g, replacement: "$1PrimaryOpen-Angle Glaucoma$2"},{pattern: / (^|[^\u4e00-\u9fa5])glaucoma([^\u4e00-\u9fa5]|$) / g, replacement: "$1POAG$2"}]. Then, the text replacement is performed using the replace method provided by the regular expression API. For example: text.replace( / (^|[^\u4e00-\u9fa5])glaucoma([^\u4e00-\u9fa5]|$) / g, "$1POAG$2").

[0078] (3) Call the server translation interface to translate and replace the text after the proper noun. The server translation will further process the initial translation in the context. For example, the long word replacement in the previous step is prioritized, but not all terms need to be expressed in full name in the context. Therefore, the translation needs to be optimized. Optimization strategies include the following aspects: ① Cross-section semantic association: by analyzing the logical relationship between adjacent text segments, ensure the consistency of pronoun translation. ② Adjust the sentence structure based on the expression habits of the target language. Finally, the server combines the optimized translation and returns it to the front-end page as the response value of the translation interface.

[0079] 4. Translation rendering. Because there is a difference in character length between the translation and the original text, if the translation is displayed on the original layer, there will be a problem of document structure disorder. For example, "organization structure" is translated into English as "organization structure", the character length increases from 4 to 22 (expansion rate is about 275%). Therefore, there will be a problem of incomplete display or structural disorder. To solve this problem, the invention uses the following steps:

[0080] (1) Copy the original text node and create a translation node copy. The text node obtained in step 2 is used as the source node (sourceNode) and cloned. By calling the DOM.cloneNode(true) method, a copy node (translationNode) is generated that is identical in structure and inline style. Then insert the copy node into the DOM tree as a sibling adjacent node of the source node.

[0081] Specifically: use sourceNode.insertAdjacentElement('afterend',translationNode) to ensure that the copy node occupies an independent rendering line. Finally, set the default display only translation, specifically, set sourceNode.style.display = 'none' to make the source node exit the rendering process but remain in the DOM.

[0082] (2) Adjust the CSS style of the copy node to complete the translation container modification. Because the setting of the CSS style of the copy node depends on the expansion rate of the translation, an expansion rate-driven adaptive style hierarchical control is set. Based on the character length expansion rate (expansionRatio) of the translation relative to the original text, the CSS style is adjusted based on the elastic adjustment strategy, specifically:

[0083] First-level strategy (expansion rate <= 100%): Use the CSS Flex box model feature to achieve elastic allocation of container space by configuring container scaling flex-grow / shrink.

[0084] Secondary strategy (Expansion rate > 100% and <= 200%): based on the expansion rate, the nonlinear scaling rate is calculated using the square root function, and then the font size is scaled according to the calculation result, and the nonlinear scaling formula is as follows:

[0085]

[0086] Wherein, ScaleFactor represents the font scaling rate, and Ratio represents the expansion rate.

[0087] Tertiary strategy (Expansion rate > 200%): Trigger automatic line break and vertical scroll bar injection to prevent layout collapse. Specifically: Set the whiteSpace attribute value to pre-wrap to allow line breaks when exceeding the container, and set the overflowY attribute value to auto.

[0088] For example: Chinese navigation menu item "Personal Center" -> English "Personal Center". The expansion rate is about 185%, and the secondary strategy is preferred, assuming the original font size is 16px, then: font-size: ceil(1 / Math.sqrt(Ratio)*fontSize)=⌈1 / ⌉=9.

[0089] (3) Bilingual rendering layer switching. By manipulating the CSS display attribute state of the source node and the copy node, the "translation only" and "source text + translation" bilingual comparison two rendering modes are realized. Specifically: First, set the display value of translationNode to block to complete the default use of translation only; second, set global listening toggleTranslation event, for example, when the user presses the tab key, the event is triggered, and the display value of the source node is switched.

[0090] 5. Continuously listen to the front-end page screenshot event, when the screenshot event is triggered, steps 1 to 4 are executed in a loop until the user closes the page or switches to the source language.

[0091] On the other hand, the embodiment of the application provides a front-end dynamic screenshot translation system for realizing the front-end dynamic screenshot translation method, and the system comprises:

[0092] Label setting module: mark the label for limiting the translation boundary in the front-end project;

[0093] The text extraction module extracts the text in the screenshot after listening to the user triggering the screenshot and uploading the screenshot snapshot, including: converting image coordinates into page coordinates to locate the position of the text in the screenshot on the page, obtaining a DOM element stack corresponding to the position, traversing the DOM element stack in combination with the hierarchical attributes of the page, reading out the stack elements to obtain candidate text nodes, performing screening on the text nodes at the edge of the screenshot based on a matching threshold, and finally obtaining the text nodes to be translated;

[0094] The target language translation module replaces the matched proper nouns based on a local terminology library, performs sorting based on a word length descending order to preferentially match and replace long words, and calls a server translation interface to translate the text after replacing the proper nouns;

[0095] The translation rendering module clones the text nodes to be translated as source nodes to generate a translation copy node that is identical in structure and inline style to the source node, inserts the copy node into the DOM tree as a sibling adjacent to the source node, adjusts the CSS style of the copy node to complete the translation container modification, and manipulates the CSS display attribute state of the source node and the copy node to realize instant switching between the translation-only mode and the bilingual comparison mode.

[0096] As to the system in the above embodiment, the specific manner in which the various modules perform operations has been described in detail in the embodiment of the method, and will not be described in detail here.

[0097] For the system embodiment, since it basically corresponds to the method embodiment, the relevant part can be referred to the part of the method embodiment. The system embodiment described above is only illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the application scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0098] Correspondingly, the application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the front-end dynamic screenshot translation method as described above. As shown in the Figure 6 The front-end dynamic screenshot translation method provided by the embodiment of the application is a hardware structure diagram of any device with data processing capability, in addition to Figure 6 In addition to the processor, the memory, and the network interface shown, any data processing capable device in which the apparatus of the embodiments is implemented can also include other hardware according to the actual functions of the data processing capable device, and no further elaboration is made.

[0099] Accordingly, the present application also provides a computer readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the front-end dynamic screenshot translation method as described above. The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or a memory, according to any of the embodiments described above. The computer readable storage medium can also be an external storage device, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. equipped on the device. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of any data processing capable device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0100] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the application be limited only by the scope of the claims, including any appropriate equivalents.

[0101] It is to be understood that the application is not limited to the precise construction hereinafter described and as shown in the attached drawings, and that various changes in form and detail can be made therein without departing from the scope of the application.

[0102] The above description is only the preferred embodiment of the present application, although the present application has been disclosed as above with the preferred embodiment, however, it is not intended to limit the present application. Any skilled person in the art, without departing from the scope of the technical solutions of the present application, can make many possible changes and modifications to the technical solutions of the present application disclosed above, or modify equivalent embodiments. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, without departing from the content of the technical solutions of the present application, still belongs to the protection scope of the technical solutions of the present application.< / translate> < / translate> < / translate> < / translate> < / translate> < / translate>

Claims

1. A front-end dynamic screenshot translation method, characterized by, The method comprises: In the front-end project, a label for defining a translation boundary is marked; A front-end page screenshot event is listened to, and when a user triggers a screenshot and uploads a screenshot snapshot, the following operations are sequentially performed: extracting text within the screenshot, target language translation, and translation rendering; The extraction of the text within the screenshot comprises: converting image coordinates into page coordinates to locate the position of the text within the screenshot on the page; obtaining a DOM element stack corresponding to the position, traversing the DOM element stack in combination with the hierarchical attributes of the page, reading out the stack elements, obtaining candidate text nodes, performing screening on the text nodes at the edge of the screenshot based on a matching threshold, and finally obtaining text nodes to be translated; the coordinate conversion formula for converting image coordinates into page coordinates is as follows: (viewportX, viewportY) = (imgX+clientX-scrollX, imgY+clientY-scrollY), wherein (viewportX, viewportY) is the positioning of the recognized text on the page, (imgX, imgY) is the image local coordinates of the text block detected by the text recognition technology, (clientX, clientY) is the starting position of the mouse pressed by the user for the screenshot, and (scrollX, scrollY) is the position of the page scrolling; the screening on the text nodes at the edge of the screenshot based on the matching threshold comprises: calculating the overlapping area of the text region corresponding to the text node at the edge of the screenshot and the screenshot region; a matching threshold is preset, and if the percentage of the overlapping area in the area of the text region exceeds the matching threshold, it is considered that the node needs to be translated, otherwise it is considered that the node does not need to be translated; The target language translation comprises: replacing matched proper nouns based on a local terminology library, and preferentially matching long words and replacing them based on a word length descending order sorting algorithm; calling a server translation interface to translate the text after replacing the proper nouns; The translation rendering comprises: cloning the text node to be translated as a source node to generate a translation copy node with the same structure and inline style, inserting the copy node into the DOM tree as a sibling adjacent node of the source node, adjusting the CSS style of the copy node to complete the translation container modification, and realizing instant switching between only translation and bilingual comparison rendering modes by manipulating the CSS display attribute state of the source node and the copy node.

2. The method of claim 1, wherein, The tags for defining the boundaries of translation include <translate>Labels and <no-translate>Labels, said <no-translate>The text in the area of the label wrap does not need to be translated, said <translate>The label adds a custom data attribute for setting attribute limits for proper nouns.< / translate> < / translate> 3. The method of claim 1, wherein, The stack elements read after traversing the DOM element stack are obtained as candidate text nodes after the following optimization processing: a. Traverse the DOM nodes and filter out the DOM nodes that do not need to be translated and are marked by the label in advance; b. Use the document.createTreeWalker() method to create a TreeWalker and specify the node type to be traversed as a text node; use depth-first node traversal, and if the next node does not exist, stop recursion; return all matched text nodes.

4. The method of claim 1, wherein, The local terminology library is associated with the data attribute for proper noun replacement configured in advance by a label; the priority matching long word includes matching the original long word and matching the translated long word, and the replacement of the proper noun includes: converting the proper noun into a regular mode with a boundary, matching all the proper nouns according to the local terminology library and performing boundary processing, and executing text replacement by using the replace method provided by the regular API.

5. The method of claim 1, wherein, According to the character length expansion rate of the translated text relative to the original text, the copy node CSS style is adjusted based on an elastic adjustment strategy, specifically: The first strategy: when the expansion rate <= 100%, the CSS Flex box model feature is used to realize the elastic distribution of container space by configuring container scaling flex-grow / shrink; The second strategy: when the expansion rate > 100% and <= 200%, a non-linear scaling rate is calculated based on the expansion rate using a square root function, and the font size is scaled according to the calculated scaling rate; The third strategy: when the expansion rate > 200%, automatic line wrapping and vertical scroll bar injection are triggered.

6. A front-end dynamic screenshot translation system, comprising: To implement the front-end dynamic screenshot translation method as claimed in any one of claims 1-5, the system comprises: A label setting module: marking a label for defining a translation boundary in a front-end project; A text extraction module: extracting text within a screenshot after listening to user triggering of screenshot and uploading of a screenshot snapshot, including: converting image coordinates to page coordinates to locate the position of the text within the screenshot on a page, obtaining a DOM element stack corresponding to the position, traversing the DOM element stack in combination with the hierarchical attribute of the page, reading out the stack elements to obtain candidate text nodes, performing filtering on the text nodes at the edges of the screenshot based on a matching threshold, and finally obtaining text nodes to be translated; A target language translation module: replacing matched proper nouns based on a local terminology library, preferentially matching long words and replacing them based on a word length descending order sorting algorithm, and calling a server translation interface to translate the text after replacing the proper nouns; A translation rendering module: cloning the text nodes to be translated as source nodes to generate translation copy nodes with the same structure and inline style, inserting the copy nodes into the DOM tree as sibling nodes adjacent to the source nodes, adjusting the CSS style of the copy nodes to complete translation container modification, and realizing instant switching between only translation and bilingual comparison rendering modes by manipulating the CSS display attribute state of the source nodes and the copy nodes.

7. An electronic device comprising a memory and a processor, characterized in that The memory is coupled with the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to realize the front-end dynamic screenshot translation method as claimed in any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the front-end dynamic screenshot translation method as claimed in any one of claims 1-5.

Images