Methods, apparatus, equipment and media for translating content displayed on a terminal screen

Abstract

This invention provides a method, apparatus, device, and storage medium for translating content displayed on a terminal screen. Related embodiments can be applied to scenarios such as mobile terminals, computers, and in-vehicle terminals. The method includes: acquiring the current display content of the terminal and performing screenshot processing to obtain an image frame to be translated, which includes text content information in the current display content; translating the text content information in the image frame to be translated to obtain a text translation result; rendering the text translation result according to text attribute information and then overlaying it onto the current display content page of the terminal. Thus, the image frame to be translated in the terminal display content can be accurately translated. Simultaneously, when presenting the text translation result, text color information, text font size information, background information, and foreground information can be rendered and overlaid onto the terminal display content, matching the visual style of the terminal display content and ensuring a more comfortable user experience.

Description

Technical Field

[0001] This invention relates to video processing technology, and more particularly to methods, apparatus, electronic devices, and storage media for translating content displayed on a terminal screen. Background Technology

[0002] With the development of machine translation, Neural Machine Translation (NMT) has become a widely used next-generation translation technology. NMT systems are built on an encoder-decoder framework. However, during the translation process, while the translated foreign text on the terminal display can accurately translate the content, the format of the translated text displayed directly on the terminal screen often doesn't match the original foreign language page's display, resulting in blurriness and ghosting, thus affecting the user experience. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for translating content displayed on a terminal screen. This not only accurately translates the text content information in the image frames to be translated within the terminal screen content, but also, when presenting the image frames to be translated, the text color information, text size information, background information, and foreground information can be rendered and overlaid onto the current display content page of the terminal, matching the visual style of the terminal screen content. This avoids ghosting and blurry text caused by translating the text content, ensuring a more comfortable user experience.

[0004] The technical solution of this invention is implemented as follows:

[0005] This invention provides a method for translating content displayed on a terminal screen, comprising:

[0006] Receive a translation trigger command to translate the current content displayed on the terminal screen;

[0007] In response to the translation processing instruction for the current screen display content of the terminal, the current screen display content of the terminal is obtained, and a screenshot is taken to obtain an image frame to be translated that includes text content information in the current screen display content of the terminal; wherein, the image frame to be translated includes text content information and text attribute information corresponding to the text content information;

[0008] The text content information in the image frame to be translated is processed to obtain the text translation result;

[0009] Obtain the text attribute information corresponding to the text content information of the image frame to be translated, wherein the text attribute information includes text color information and text font size information;

[0010] After rendering the text translation result based on the text attribute information, it is overlaid onto the current screen display page of the terminal to achieve real-time presentation of the rendered text translation result on the terminal.

[0011] This invention also provides an apparatus for translating content displayed on a terminal screen, comprising:

[0012] The information transmission module is used to receive translation trigger commands to translate the content currently displayed on the terminal screen;

[0013] The information processing module is used to respond to the translation processing instruction for the current screen display content of the terminal, obtain the current screen display content of the terminal, and perform screen screenshot processing to obtain an image frame to be translated that includes text content information in the current screen display content of the terminal; wherein, the image frame to be translated includes text content information and text attribute information corresponding to the text content information;

[0014] The information processing module is used to translate the text content information in the image frame to be translated, and obtain the text translation result;

[0015] The information processing module is used to obtain text attribute information corresponding to the text content information of the image frame to be translated, wherein the text attribute information includes text color information and text font size information;

[0016] The information transmission module renders the text translation result according to the text attribute information and then overlays it onto the current screen display page of the terminal, so as to realize the real-time presentation of the rendered text translation result on the terminal.

[0017] In the above scheme,

[0018] The information processing module is used to trigger the corresponding translation model in response to the translation processing instruction for the current screen display content of the terminal.

[0019] The information processing module is used to determine at least one word-level latent variable corresponding to the text content information through the encoder of the translation model.

[0020] The information processing module is used to generate, through the decoder of the translation model, translation words corresponding to the word-level latent variables and the selection probability of the translation words, based on the at least one word-level latent variable;

[0021] The information processing module is used to select at least one translated word to form a text translation result corresponding to the text content information based on the selection probability of the translation result.

[0022] In the above scheme,

[0023] The information processing module is used to convert the format of the image frame to be translated to obtain a bitmap format image frame to be translated.

[0024] The information processing module is used to perform color space conversion processing on the image frame to be translated in the image format, and to perform identification processing on the background color in the image frame to be translated based on the result of the color space conversion processing, so as to obtain the color value of the pixel corresponding to the background color.

[0025] The information processing module is used to extract the color values ​​of the pixels in the region where the text content information in the image frame to be translated is located;

[0026] The information processing module is used to compare the color value of the pixel corresponding to the background color with the color value of the pixel in the area where the text content information is located, and obtain the text color information.

[0027] The information processing module is used to obtain the text font size information by segmenting the bitmap format image frame to be translated.

[0028] In the above scheme,

[0029] The information processing module is used to segment the image frame to be translated based on the position of the text content information to obtain at least two text recognition segments.

[0030] The information processing module is used to count the text size corresponding to the single character involved in the text of the two or fewer text recognition segments to obtain the text size information in each text recognition segment.

[0031] The information processing module is used to compare the text size information in different text recognition segments. When it is determined that the horizontal starting position of different text recognition segments is less than or equal to the horizontal position threshold and the height of different text recognition segments is less than or equal to the height threshold, the different text recognition segments are summarized.

[0032] The information processing module is used to statistically analyze the text size information of the text recognition segments obtained after summary processing, and to filter the corresponding text size information in the image frame to be translated based on the statistical results.

[0033] In the above scheme,

[0034] The information processing module is used to determine the background and foreground information corresponding to the text content information based on the position of the text content information in the image frame to be translated;

[0035] The information processing module is used to perform matching processing on the text translation result according to the text color information and text font size information in the text attribute information to obtain a first text translation result, wherein the first text translation result matches the screen style of the current screen display content;

[0036] The information processing module is used to render the first text translation result based on the background information and foreground information corresponding to the text content information, so as to obtain the rendered text translation result.

[0037] In the above scheme,

[0038] The information processing module is used to convert the image frame to be translated from the RGB mode to the HSV mode.

[0039] The information processing module is used to perform cropping processing on the text content information in the image frame to be translated in the HSV mode, according to the position of the text content information in the image frame to be translated.

[0040] The information processing module is used to determine the background information corresponding to the text content information based on the result of the cropping process.

[0041] The information processing module is used to determine the edge position of the text content information based on the result of the cropping process and the position of the text content information in the image frame to be translated, and to determine the foreground information based on the edge position of the text content information.

[0042] In the above scheme,

[0043] The information processing module is used to configure a text region boundary detection box and a maximum search radius based on the background information corresponding to the text content information.

[0044] The information processing module is used to perform a first marking process on pixels that exceed the text region boundary detection box based on the text region boundary detection box and the maximum search radius, and obtain a first marking result;

[0045] The information processing module is used to perform a second marking process on the pixels inside the text region boundary detection box to obtain a second marking result;

[0046] The information processing module is used to traverse all the pixels corresponding to the text content information in the two-marking results and perform pixel reverse filling processing to obtain the background processing result of the first text translation result;

[0047] The information processing module is used to perform foreground information fusion processing in response to the background processing result of the first text translation result, so as to obtain the fused first text translation result.

[0048] In the above scheme,

[0049] The information processing module is used to perform Gaussian blur processing on the image of the area where the foreground information is located;

[0050] The information processing module is used to convert the result of the Gaussian blurring process from RGB mode to HSV mode to obtain foreground information in HSV mode.

[0051] The information processing module is used to sequentially perform morphological dilation and morphological erosion processing on the image of the region where the foreground information is located, to obtain the first text translation result after fusion.

[0052] In the above scheme,

[0053] The information processing module is used to obtain a replacement operation for the target bullet screen through the bullet screen editing component when the terminal screen displays a game video and the text content information is a bullet screen.

[0054] The information processing module is used to respond to the replacement operation and obtain the second text translation result corresponding to the target bullet screen;

[0055] The information processing module is used to render the second text translation result according to the text attribute information through the bullet screen editing component;

[0056] The information processing module is used to replace the target bullet screen in the current screen display content page of the terminal with the second text translation result after rendering.

[0057] In the above scheme,

[0058] The information processing module is used to control and adjust the display position of the new bullet screen corresponding to the control layer component of the current screen display content of the terminal, so as to realize the display of the new bullet screen.

[0059] In the above scheme,

[0060] The information processing module is used to determine an empirical threshold that matches the image frame to be translated when the content displayed on the terminal screen is a cloud gaming video.

[0061] The information processing module is used to determine the overflow pixels in the terminal screen display content based on an empirical threshold matching the image frame to be translated and a boundary range of image processing matching the color key parameters;

[0062] The information processing module is used to adjust the grayscale value of the overflow pixels in the content displayed on the terminal screen.

[0063] In the above scheme,

[0064] The information processing module is used to present a translation function item in the view interface, and the translation function item is used to translate the text information in the content displayed on the terminal screen;

[0065] The information processing module is configured to, in response to a trigger operation for the translation function item, acquire and present the translated image frame and the corresponding background image after translation processing, wherein the translated image frame after translation processing includes the first text translation result.

[0066] In the above scheme,

[0067] The information processing module is used to present a translation function item in the form of a floating ball in the view interface, wherein the translation function item in the form of a floating ball can adjust its display position in the view interface in response to the trigger operation; or...

[0068] The information processing module is used to present a transparent translation function item in the view interface, wherein the number of pixels of the transparent translation function item is 1, and the transparent translation function item can adjust its display position in the view interface in response to the trigger operation.

[0069] This invention also provides an electronic device, the electronic device comprising:

[0070] Memory, used to store executable instructions;

[0071] A processor, when executing executable instructions stored in the memory, implements a method for translating the content displayed on the terminal screen.

[0072] This invention also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement a method for translating content displayed on a terminal screen.

[0073] The embodiments of the present invention have the following beneficial effects:

[0074] The system receives a translation trigger command to translate the currently displayed content on the terminal. In response to the translation command, it acquires the current displayed content and performs a screenshot to obtain an image frame containing text content information to be translated. The image frame includes text content information and corresponding text attribute information. The system translates the text content information in the image frame to obtain a text translation result. It acquires the text attribute information corresponding to the text content information of the image frame, including text color and font size information. Based on the text attribute information, it renders the text translation result and overlays it onto the current displayed content page of the terminal, thereby achieving real-time presentation of the rendered text translation result on the terminal. Therefore, it can not only accurately translate the image frames to be translated in the terminal screen content, but also, when presenting the image frames to be translated, the text color information, text font size information, background information and foreground information can be rendered and overlaid on the current screen content page of the terminal, matching the screen style of the terminal screen content, avoiding ghosting and blurry text caused by directly translating the text content, and ensuring that users have a more comfortable user experience. Attached Figure Description

[0075] Figure 1 This is a schematic diagram illustrating a usage scenario of the method for translating content displayed on a terminal screen provided in an embodiment of the present invention;

[0076] Figure 2 A schematic diagram of the composition structure of an electronic device provided in an embodiment of the present invention;

[0077] Figure 3A This is an optional flowchart illustrating a method for translating content displayed on a terminal screen, as provided in an embodiment of the present invention.

[0078] Figure 3B This is a schematic diagram of the image frame to be translated in the method for translating content displayed on a terminal screen provided in an embodiment of the present invention.

[0079] Figure 4 This is a schematic diagram of an optional processing interface for a method of translating content displayed on a terminal screen provided in an embodiment of the present invention.

[0080] Figure 5 This is the structure of the display layer component in an embodiment of the present invention;

[0081] Figure 6 This is a schematic diagram of an optional processing interface for a method of translating content displayed on a terminal screen provided in an embodiment of the present invention.

[0082] Figure 7 This is a schematic diagram of an optional processing interface for a method of translating content displayed on a terminal screen provided in an embodiment of the present invention.

[0083] Figure 8 This is a schematic diagram of an optional processing interface for a method of translating content displayed on a terminal screen provided in an embodiment of the present invention.

[0084] Figure 9 This is a schematic diagram illustrating the processing of target bullet comments by the method for translating terminal screen content provided in an embodiment of the present invention.

[0085] Figure 10 This is a schematic diagram illustrating the method for translating terminal screen content to form new bullet comments, as provided in an embodiment of the present invention.

[0086] Figure 11 This is a schematic diagram illustrating the display effect of a new bullet screen generated by the method for translating content displayed on a terminal screen according to an embodiment of the present invention.

[0087] Figure 12 This is an optional flowchart illustrating a method for translating content displayed on a terminal screen, as provided in an embodiment of the present invention.

[0088] Figure 13 This is an optional flowchart illustrating a method for translating content displayed on a terminal screen, as provided in an embodiment of the present invention.

[0089] Figure 14 This is an optional flowchart illustrating a method for translating content displayed on a terminal screen, as provided in an embodiment of the present invention. Detailed Implementation

[0090] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0091] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0092] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0093] 1) Neural Network (NN): Artificial Neural Network (ANN), also known as neural network or neural network-like network, is a mathematical or computational model in the fields of machine learning and cognitive science that imitates the structure and function of biological neural networks (the central nervous system of animals, especially the brain) and is used to estimate or approximate functions.

[0094] 2) Speech Recognition (SR): Also known as Automatic Speech Recognition (ASR), Computer Speech Recognition (CSR), or Speech to Text (STT), its goal is to use computers to automatically convert human speech into corresponding text.

[0095] 3) Machine Translation (MT): This falls under the category of computational linguistics, and its research focuses on using computer programs to translate text or speech from one natural language into another. Neural Machine Translation (NMT) is a technique that uses neural networks for machine translation.

[0096] 4) Speech Translation: Also known as automatic speech translation, it is a technology that uses computers to translate the speech of one natural language into the text or speech of another natural language. It generally consists of two stages: speech recognition and machine translation.

[0097] 5) Optical Character Recognition (OCR) converts text from various tickets, newspapers, books, manuscripts and other printed materials into image information through optical input methods such as scanning, and then uses character recognition technology to convert the image information into usable computer input technology.

[0098] 6) In response to, used to indicate the conditions or states on which the operation performed depends. When the conditions or states on which it depends are met, one or more operations performed may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0099] 7) Virtual Scene: This is a virtual scene displayed (or provided) by an application when it runs on a terminal. This virtual scene can be a simulation environment of the real world, a semi-simulated and semi-fictional three-dimensional environment, or a purely fictional three-dimensional environment.

[0100] The virtual scene can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene. The following embodiments use a three-dimensional virtual scene as an example, but this is not a limitation. Optionally, the virtual scene can also be used for virtual scene battles between at least two virtual objects. Optionally, the virtual scene can also be used for battles between at least two virtual objects using virtual weapons. Optionally, the virtual scene can also be, but is not limited to, shooting games, parkour games, racing games, multiplayer online battle arena (MOBA), racing games (RCG), and sports games (SPG). Using the trained data processing model provided in this application, it can be deployed in the game servers corresponding to the aforementioned game scenes to generate real-time virtual scene movement routes and present them in the game interface, execute corresponding actions in the corresponding games, simulate the operation of virtual users, and jointly complete different types of games in the virtual scene with actual users participating in the game.

[0101] 8) Virtual objects: These are interactive images of people and objects within a virtual scene, or movable objects within the virtual scene. These movable objects can be virtual characters, virtual animals, cartoon characters, etc., such as people, animals, plants, oil drums, walls, stones, etc., displayed in the virtual scene. A virtual object can be a virtual avatar representing the user within the virtual scene. A virtual scene can include multiple virtual objects, each with its own shape and volume, occupying a portion of the space within the virtual scene.

[0102] Figure 1 This is a schematic diagram illustrating a usage scenario of the method for translating content displayed on a terminal screen provided in an embodiment of the present invention. (See also...) Figure 1 The terminal (including terminal 10-1 and terminal 10-2) is equipped with a client for video playback. The terminal connects to server 200 through network 300. Network 300 can be a wide area network or a local area network, or a combination of the two, and data transmission is achieved using a wireless link.

[0103] As an example, the terminals (terminal 10-1 and / or terminal 10-2) are used to obtain the video and translation results from the server 200, and display the text translation results in the video being played to help users understand the video content.

[0104] The terminal (including terminal 10-1 and terminal 10-2) is also used to save the newly formed text translation result in the corresponding storage medium after obtaining the editing operation for the target text translation result through the text translation result editing component and forming a new text translation result based on the content of the target text translation result.

[0105] The terminal (terminal 10-1 and / or terminal 10-2) is also used to obtain the new text translation results stored in the server 200 via the network 300, and to display all text translation results returned by the server 200, including the new text translation results, to the user during the playback of the current screen content of the terminal.

[0106] The structure of the electronic device according to the embodiments of the present invention will be described in detail below. The electronic device can be implemented in various forms, such as terminals such as smartphones, tablets and desktop computers with video playback functions, or servers with video display functions. Figure 2 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of the present invention. It can be understood that... Figure 2 The diagram shows only an exemplary structure of the electronic device, not all of it; implementation is possible as needed. Figure 2 The structure shown may be part or all of the structure.

[0107] The electronic device provided in this embodiment of the invention includes at least one processor 201, a memory 202, a user interface 203, and at least one network interface 204. The various components in the electronic device 20 are coupled together via a bus system 205. It is understood that the bus system 205 is used to implement communication between these components. In addition to a data bus, the bus system 205 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 205.

[0108] The user interface 203 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0109] It is understood that memory 202 can be volatile memory or non-volatile memory, or both. In this embodiment of the invention, memory 202 is capable of storing data to support the operation of a terminal (such as 10-1). Examples of this data include any computer programs used to operate on the terminal (such as 10-1), such as operating systems and applications. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications.

[0110] In some embodiments, the apparatus for translating terminal screen display content provided in this invention can be implemented using a combination of hardware and software. For example, the image processing device provided in this invention can be a processor in the form of a hardware decoding processor, programmed to execute the method for translating terminal screen display content provided in this invention. For instance, the processor in the form of a hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0111] As an example of the hardware and software combined implementation of the device for translating terminal screen display content provided in this embodiment of the invention, the device for translating terminal screen display content provided in this embodiment of the invention can be directly embodied as a software module executed by processor 201. The software module can be located in a storage medium, which is located in memory 202. Processor 201 reads the executable instructions included in the software module in memory 202 and combines them with necessary hardware (e.g., including processor 201 and other components connected to bus 205) to complete the method for translating terminal screen display content provided in this embodiment of the invention.

[0112] As an example, processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0113] As an example of the hardware implementation of the device for translating terminal screen display content provided in this embodiment of the invention, the device provided in this embodiment of the invention can be directly executed by a processor 201 in the form of a hardware decoding processor. For example, it can be executed by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the method for translating terminal screen display content provided in this embodiment of the invention.

[0114] In this embodiment of the invention, the memory 202 is used to store various types of data to support the operation of the electronic device 20. Examples of such data include: any executable instructions for operation on the electronic device 20, such as executable instructions that implement the method for translating content displayed on a terminal screen according to this embodiment of the invention, which may be included in the executable instructions.

[0115] In other embodiments, the apparatus for translating terminal screen display content provided in this invention can be implemented in software. Figure 2 An apparatus 2020 for translating terminal display content, stored in memory 202, is shown. This apparatus can be software in the form of programs and plugins, and includes a series of modules. As an example of a program stored in memory 202, it may include the apparatus 2020 for translating terminal display content. The apparatus 2020 includes the following software modules: an information transmission module 2081, an information processing module 2082, and a data layer module 2083. When the software modules in the apparatus 2020 for translating terminal display content are read into RAM and executed by processor 201, the method for translating terminal display content provided in this embodiment of the invention will be implemented. The functions of each software module in the apparatus 2020 for translating terminal display content will be described below.

[0116] Information transmission module 2081 is used to receive translation trigger commands for translating the current screen display content of the terminal;

[0117] The information processing module 2082 is used to respond to the translation processing instruction for the current screen display content of the terminal, obtain the current screen display content of the terminal, and perform screen screenshot processing to obtain an image frame to be translated that includes text content information in the current screen display content of the terminal; wherein, the image frame to be translated includes text content information and text attribute information corresponding to the text content information;

[0118] The information processing module 2082 is used to translate the text content information in the image frame to be translated to obtain the text translation result.

[0119] The information processing module 2082 is used to render the text translation result according to the text attribute information and then overlay it onto the current screen display content page of the terminal, so as to realize the real-time presentation of the rendered text translation result on the terminal.

[0120] according to Figure 2 The electronic device shown, in one aspect of this application, also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform various alternative implementations and combinations thereof of the above-described method for translating content displayed on a terminal screen.

[0121] The method for translating terminal screen content provided in this invention is described in conjunction with the exemplary application and implementation of the terminal provided in the embodiments of the present invention. As can be understood from the above, the method for translating terminal screen content provided in the embodiments of the present invention can be performed by various types of devices with video processing functions, such as dedicated video playback devices, computers, and servers.

[0122] Combination Figure 2 The illustrated electronic device 20 illustrates the method for translating terminal screen display content provided by an embodiment of the present invention. See also... Figure 3A , Figure 3A This is an optional flowchart illustrating a method for translating content displayed on a terminal screen, as provided in an embodiment of the present invention. It can be understood that... Figure 3A The steps shown can be performed by various electronic devices that run video processing functions, such as terminals like computers and smartphones with video playback and processing capabilities, servers, or server clusters. The following section addresses... Figure 3A The steps shown are explained.

[0123] Step 301: Receive a translation trigger command to translate the content currently displayed on the terminal screen.

[0124] In some embodiments of the present invention, the terminal includes, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, and in-vehicle terminals. Taking the text information in the display interface of a game program as an example, by acquiring real-time screenshots of the user's screen during the game, and combining the translation process provided by the game server to translate the game screenshots in real time, a text translation result is generated. Finally, after rendering processing, it is overlaid onto the original screen interface, allowing the user to understand the meaning of the text information displayed in the game video.

[0125] In some embodiments, when using the method for translating terminal screen display content provided by the present invention, a translation trigger instruction for translating the current terminal screen display content can be received by presenting a translation function item in the view interface; wherein, the presented translation function item can always be in the active state, and when the terminal screen is turned on, it can provide real-time monitoring of the terminal screen display content, and perform screen capture processing in response to the translation processing instruction for the current terminal screen display content to obtain an image frame to be translated that includes text content information in the current terminal screen display content;

[0126] In some embodiments, when the user of the terminal does not authorize the information collection of the presented translation function, the presented translation function can be in a closed state. By triggering the presented translation function, a translation trigger instruction is received to translate the current screen display content of the terminal. Then, in response to the translation processing instruction for the current screen display content of the terminal, the current screen display content of the terminal is obtained, and a screenshot is taken to obtain an image frame to be translated that includes text content information in the current screen display content of the terminal. After the rendered text translation result is presented in real time on the terminal, the presented translation function is adjusted to a closed state to ensure the security of user information of the terminal.

[0127] refer to Figure 3B , Figure 3B This is a schematic diagram of the image frame to be translated in the method for translating content displayed on a terminal screen provided in an embodiment of the present invention. A translation function 3001 can be presented in the view interface. The translation function 3001 is used to translate text information in the content displayed on the terminal screen. In response to a trigger operation on the translation function 3001, the translated image frame and the corresponding background image are acquired and presented. The translated image frame includes the corresponding text translation result.

[0128] like Figure 3BAs shown, the translation function item 3001 appears as a floating ball in the view interface. Upon receiving a control command, the floating ball-shaped translation function item can adjust its display position in the view interface in response to the trigger operation. In some embodiments, the translation function item 3001 can also appear as a transparent form in the view interface. The transparent form of the translation function item has 1 pixel. It can adjust its display position in the view interface in response to the trigger operation. By adjusting the number of pixels in the translation function item 3001 to 1, the transparent form can avoid obscuring the text in the game video displayed in the view interface, thus achieving accurate translation of the text information in the image frame to be translated. As an example, the transparent form of the translation function item can be set to a floating ball shape by default. Users can also adjust the shape of the translation function item 3001 according to their needs. This embodiment of the invention does not impose specific limitations on this.

[0129] In some embodiments of the present invention, the method for translating terminal screen display content provided by the present invention can also process foreign language bullet screen information in long videos, realizing the translation and display of foreign language bullet screen information. Specifically, when the video client running on the terminal obtains video data sent by the corresponding server, it can simultaneously obtain the text translation result sent by the server and store it in the queue of text translation results to be displayed corresponding to the obtained video. The text translation result obtained by the client through the current screen display content of the terminal can be submitted by different users watching the same video through their respective clients, or it can be submitted by different users watching the same video through the corresponding web page playback interface. When playing the current screen display content of the terminal through the client, the queue of text translation results to be displayed is called to obtain the second text translation result corresponding to the target bullet screen. The bullet screen editing component renders the second text translation result according to the text attribute information. The rendered second text translation result replaces the target bullet screen in the current screen display content page of the terminal, so that users can understand the foreign language bullet screen information in a timely manner.

[0130] In some embodiments of the present invention, the display of text translation results during video playback by a video client (e.g., a video player) running on a terminal can be achieved in the following way: the video is played through the screen layer of the client, and the text translation results in the queue of text translation results to be displayed are displayed through a text translation result layer that floats above the screen layer.

[0131] In some embodiments of the present invention, when the text content information to be translated is the bullet screen information of a game video, during the playback of the video by the video client running on the terminal, the text translation results in the queue of text translation results to be displayed enter from one side of the playback area of ​​the current screen display content of the terminal and move to the other side. When they move to the other side of the video playback area, the display stops. In this invention, no specific limitation is made on the movement method of the text translation results.

[0132] In some embodiments of the present invention, displaying the text translation result in the current screen display content of the terminal includes:

[0133] The control layer component corresponding to the current screen display content of the terminal controls and adjusts the display position of the text translation result to achieve the display of the text translation result. Specifically, the control layer component can adjust the position, dwell time, and display position of the text translation result in the video playback area from the queue of text translation results to be displayed.

[0134] Step 302: In response to the translation processing instruction for the current screen display content of the terminal, obtain the current screen display content of the terminal, and perform screen screenshot processing to obtain the image frame to be translated, which includes text content information in the current screen display content of the terminal.

[0135] The image frame to be translated shown in step 302 includes text content information and text attribute information corresponding to the text content information. Subsequent embodiments will continue to describe the processing of text content information and the rendering process based on text attribute information.

[0136] In some embodiments of the present invention, when a video client running on a terminal starts playing the content currently displayed on the terminal screen, the acquisition of the selection operation for the displayed text translation result is triggered.

[0137] In some embodiments of the present invention, when the client or game client of the current screen display content of the terminal is launched, the acquisition of the selection operation of the translation result of the displayed text content information is triggered simultaneously, so as to maintain the acquisition state of the selection operation of the translation result of the displayed text during the playback of the current screen display content of the terminal.

[0138] In some embodiments of the present invention, the selection operation for the displayed text translation result includes:

[0139] When the terminal displays the corresponding text translation result in the current screen content, the presentation layer component corresponding to the current screen content acquires the image frame of the video playback state, performs screenshot processing, and forms different bitmaps. Each pixel in the bitmap is assigned a specific position and color value. The color information of each pixel is represented by an RGB combination or a grayscale value.

[0140] Based on bit depth, bitmaps can be categorized into 1-bit, 4-bit, 8-bit, 16-bit, 24-bit, and 32-bit images. The more bits of information used per pixel, the more colors are available, resulting in more realistic color representation, but also a larger data volume. For example, a pixel bitmap with a bit depth of 1 has only two possible values ​​(black and white), hence it is also called a binary bitmap. An image with a bit depth of 8 has 2^8 (256) possible values. A grayscale image with a bit depth of 8 has 256 possible gray values. Since the content displayed on the terminal screen can be either game videos or long videos, bitmaps can be acquired through different methods. For example, when the electronic device is a touch-screen device with video playback capabilities, the selection operation can be achieved through touch control of the electronic device's display interface; when the electronic device is a video playback device connected to an external operating device, the selection operation can be performed by the external operating device, which includes, but is not limited to, a mouse, keyboard, and joystick.

[0141] Step 303: Translate the text content information in the image frame to be translated to obtain the text translation result.

[0142] The text content information in the image frame to be translated can be processed to obtain the text translation result, which can be achieved in the following way:

[0143] In response to the translation processing instruction for the current screen display content of the terminal, the corresponding translation model is triggered; the encoder of the translation model determines at least one word-level latent variable corresponding to the text content information; the decoder of the translation model generates translation words corresponding to the word-level latent variables and the selection probability of the translation words based on the at least one word-level latent variable; based on the selection probability of the translation result, at least one translation word is selected to form a text translation result corresponding to the text content information.

[0144] In some embodiments of the present invention, since the current screen display content of a foreign language terminal may have the problem that the text content information may be a long sentence (for example, when a Japanese server game program is executed, the current screen display content of the terminal is: 私のを打ち破ることができなくて私を更に強大にならせます), therefore, a splitting operation for the target text translation result can also be obtained through a text translation result editing component to form at least one splitting point; according to the at least one splitting point, the text content constituting the new text translation result is screened from the target text translation result; the screened text content is combined to form the new text translation result to adapt to the translation of a longer sentence.

[0145] Continue to refer to Figure 4 , Figure 4 FIG. is an optional structural schematic diagram of a translation model in an embodiment of the present invention. Among them, the Encoder includes: N = 6 identical layers, and each layer contains two sub-layers. The first sub-layer is the multi-head attention layer, and then a simple fully connected layer. A residual connection and normalization are added to each sub-layer.

[0146] The decoder consists of N=6 identical layers, which are not identical to the encoder. Each layer contains three sub-layers: a self-attention layer, an encoder-decoder attention layer, and a fully connected layer. The first two sub-layers are based on multi-head attention layers. Specifically, Nx on the left represents the structure of one layer in the encoder, which includes two sub-layers: a multi-head attention layer and a forward propagation layer. The input and output of each sub-layer are correlated; the output of the current sub-layer serves as the input for the next. Each sub-layer is followed by a normalization operation, which improves the model's convergence speed. Nx on the right represents the structure of one layer in the decoder, which includes three sub-layers. The first sub-layer is a multi-head attention sub-layer controlled by a mask matrix, used to model the generated target sentence vector. During training, a mask matrix is ​​needed to control the multi-head attention calculation, ensuring that only the first t-1 words are calculated at each multi-head attention iteration. The second sub-layer is the multi-head attention sub-layer, which is the attention mechanism between the encoder and decoder. It searches for relevant semantic information in the source text, and this layer is calculated using a dot product. The third sub-layer is the forward propagation sub-layer, calculated in the same way as the forward propagation sub-layer in the encoder. Each sub-layer of the decoder is also interconnected; the output of the current sub-layer serves as the input data for the next sub-layer. Furthermore, each sub-layer of the decoder is immediately followed by a normalization operation to accelerate model convergence.

[0147] Continue to refer to Figure 5 , Figure 5This is an optional schematic diagram of the translation process of the translation model in an embodiment of the present invention. Here, both the encoder and decoder parts contain 6 encoders and decoders. The inputs entering the first encoder are combined with the embedding and positional embedding. After passing through 6 encoders, the output is sent to each decoder in the decoder part; the input sentence to be translated is in Japanese: "无双の魔呂布は言った。「これから戦場は私一人で支配します。私と戦う人がいます。蒼天翔竜趙雲は言います。「勇者の誓いは、生死よりも甚だしい!恐れないと心に抱いて、空を飛べます。傲慢な炎の銃のあの軽食は言います:“私のを打ち破ることができなくて、私を更に強大にならせます!”剣聖宮本武蔵は「天下無双!秘密を教えます。私は無敵です。」” After being processed by the translation model, the output translation result is: "The无双魔 Lü Bu said: "From now on, the battlefield will be dominated by me alone! Is there anyone who dares to fight me?" The Azure Sky Soaring Dragon Zhao Yun said: "The oath of a brave man is more important than life and death! With fearlessness in heart, one can soar through the sky!" The arrogant Flame Spear Nezha said: "Those who cannot defeat me will make me even stronger!" The Sword Saint Miyamoto Musashi said: "Peerless in the world! Let me tell you a secret: I am invincible."".

[0148] Continue to refer to Figure 6 , Figure 6 This is an optional schematic diagram of the encoder in the translation model in an embodiment of the present invention. Here, its input consists of a query (Q) and a key (K) with dimension d and a value (V) with dimension d. All keys calculate the dot product of the query and apply the softmax function to obtain the weights of the values.

[0149] Continue to refer to Figure 6 , Figure 6 This is the vector schematic diagram of the encoder in the translation model in an embodiment of the present invention. Here, Q, K, and V are obtained by multiplying the vector x input to the encoder with W^Q, W^K, and W^V. The dimensions of W^Q, W^K, and W^V in the article are (512, 64), and then assuming that the dimension of the inputs can be (m, 512), where m represents the number of words. So the dimensions of Q, K, and V obtained after multiplying the input vector with W^Q, W^K, and W^V are (m, 64).

[0150] Continue to refer to Figure 7 , Figure 7This is a schematic diagram of vector concatenation in the encoder of the translation model in this embodiment of the invention. Z0 to Z7 are the corresponding 8 parallel heads (with dimensions of (m, 64)). Concatenating these 8 heads yields a dimension of (m, 512). Finally, multiplying by W^O results in an output matrix with a dimension of (m, 512), which then matches the dimension of the next encoder.

[0151] Continue to refer to Figure 8 , Figure 8 This diagram illustrates the encoding process of the encoder in the translation model of this invention. x1 reaches the state z1 through self-attention. The tensor after self-attention needs to undergo processing by a residual network and Later Norm, before entering a fully connected feedforward network. The feedforward network performs the same residual processing and normalization. Finally, the output tensor enters the next encoder. This process iterates six times, and the result of the iterations enters the decoder.

[0152] Continue to refer to Figure 9 , Figure 9 This is a schematic diagram of the decoding process of the decoder in the translation model of this invention, where the decoder's input and output and the decoding process are shown below:

[0153] Output: The probability distribution of the output word at position i;

[0154] Input: the encoder's output & the decoder's output at position i-1. Therefore, the attention mechanism in the middle is not self-attention; its K and V come from the encoder, and its Q comes from the decoder's output at the previous position.

[0155] The method for translating terminal screen content provided in this application is based on artificial intelligence (AI). AI is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making functions.

[0156] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0157] In the embodiments of this application, the main artificial intelligence software technologies involved include the aforementioned speech processing technologies and machine learning. For example, it may involve Automatic Speech Recognition (ASR) technology in speech technology, including speech signal preprocessing, speech signal frequency analyzing, speech signal feature extraction, speech signal feature matching / recognition, and speech training.

[0158] For example, this could involve machine learning (ML), a multidisciplinary field encompassing probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning typically includes techniques such as deep learning, which includes artificial neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and deep neural networks (DNNs).

[0159] Step 304: Obtain the text attribute information corresponding to the text content information of the image frame to be translated, wherein the text attribute information includes text color information and text font size information.

[0160] In some embodiments of the present invention Figure 10 This is an optional flowchart illustrating a method for translating content displayed on a terminal screen, as provided in an embodiment of the present invention. It can be understood that... Figure 10 The steps shown can be performed by various electronic devices that run video processing functions, such as terminals like computers and smartphones with video playback and processing capabilities, servers, or server clusters. The following section addresses... Figure 10 The steps shown are explained.

[0161] Step 1001: Convert the format of the image frame to be translated to obtain a bitmap format image frame to be translated.

[0162] Step 1002: Perform color space conversion processing on the image frame to be translated in the image format, and according to the result of the color space conversion processing, perform identification processing on the background color in the image frame to be translated to obtain the color value of the pixel corresponding to the background color.

[0163] The method for converting the color space of the bitmap image frame to be translated from RGB to HSV color space is as follows: where (r, g, b) are the red, green and blue color values ​​of a color, respectively, and their values ​​are real numbers between 0 and 1; (h, s, v) are the hue, saturation and lightness, respectively. The transformation process is referred to Formula 1.

[0164] Formula 1

[0165] in, , .

[0166] To avoid the text being affected by the background, the background of the text is designed by default to be relatively similar in color, avoiding significant color differences. Therefore, when identifying the main background color, simply take the HSV color values ​​at positions (0, 0) and (width, height) in the area near the text.

[0167] Furthermore, one can try comparing the HSV color values ​​of the main background color and other pixels in the text area image to obtain the color difference value. The method for calculating the difference value is shown below, where d represents the color difference calculation formula, and col1 and col2 represent the abbreviations of the (h, s, v) values ​​of the HSV color space of the two pixels, respectively. The transformation process is referenced in Formula 2.

[0168]

[0169] Formula 2

[0170]

[0171]

[0172] By sampling and traversing all pixels using the method described above, the pixel with the largest difference from the set main background color can be identified as the font color of the text in the image frame to be translated.

[0173] Step 1003: Extract the color values ​​of the pixels in the region where the text content information is located in the image frame to be translated;

[0174] Step 1004: Compare the color value of the pixel corresponding to the background color with the color value of the pixel in the area where the text content information is located to obtain the text color information;

[0175] Step 1005: Obtain the text font size information by segmenting the bitmap format image frame to be translated.

[0176] In some embodiments of the present invention, in order to obtain font size information, in some embodiments of the present invention, Figure 11 An optional flowchart illustrating the method for translating terminal screen display content provided in this embodiment of the invention specifically includes:

[0177] Step 1101: Based on the location of the text content information, segment the image frame to be translated to obtain at least two text recognition segments.

[0178] For example, multiple lines of text in a list may have the same font size and the same number of characters in the foreign language. These texts are typically recognized by the OCR engine as multiple text regions, translated separately, and the results returned. If the font size of these list-type or grid-type texts is calculated separately at this point, the inconsistent character lengths in the translation results may lead to different font sizes being calculated. Ultimately, multiple text regions with the same font size in the original text may be translated into multiple text regions with different font sizes. At least two text recognition segments can be segmented into different text recognition segments to achieve accurate font size recognition.

[0179] Step 1102: Statistically count the text size corresponding to the single character involved in the two or fewer text recognition segments to obtain the text size information in each text recognition segment;

[0180] Step 1103: Compare the text size information in different text recognition segments. When it is determined that the horizontal starting position of different text recognition segments is less than or equal to the horizontal position threshold, and the height of the different text recognition segments is less than or equal to the height threshold, the different text recognition segments are summarized.

[0181] like Figure 11 As shown, the text area is first aggregated and the font size is calculated uniformly. The conditions for aggregating text fragments A1, A2, etc., are as follows:

[0182] 1. The horizontal starting positions of regions A1 and A2 are equal or similar;

[0183] 2. Regions A1 and A2 have the same or similar elevation;

[0184] For each summary region, the maximum font size that can accommodate the translated text can be calculated for each region based on its width. Then, the minimum value is taken from the maximum font size results for the A-type regions to ensure that each A-type text fragment can be displayed completely using this font size.

[0185] Step 1104: Statistically analyze the text size information of the text recognition segments obtained after the summary processing, and filter the corresponding text size information in the image frame to be translated based on the statistical results.

[0186] Step 305: After rendering the text translation result according to the text attribute information, overlay it onto the current screen display content page of the terminal to realize the real-time presentation of the rendered text translation result on the terminal.

[0187] In some embodiments of the present invention Figure 12 This is an optional flowchart illustrating a method for translating content displayed on a terminal screen, provided in an embodiment of the present invention. Figure 11 After obtaining the corresponding text font size information in the image frame to be translated through the processing shown, it is necessary to... Figure 12 The steps shown involve rendering the text translation result based on the text color and font size information included in the text attribute information, and then overlaying it onto the current screen display page of the terminal. Specifically, these steps include:

[0188] Step 1201: Determine the background and foreground information corresponding to the text content information based on the position of the text content information in the image frame to be translated.

[0189] In some embodiments of the present invention, in order to obtain foreground information, the to-be-translated image frame may be converted from the Red-Green-Blue (RGB) mode to the Hue-Saturation-Value (HSV) mode; in the to-be-translated image frame in the HSV mode, the text content information in the to-be-translated image frame is cropped according to the position of the text content information in the to-be-translated image frame; according to the result of the cropping process, the background information corresponding to the text content information is determined; according to the result of the cropping process and the position of the text content information in the to-be-translated image frame, the edge position of the text content information is determined, and the foreground information is determined according to the edge position of the text content information.

[0190] Step 1202: According to the text color information and text font size information in the text attribute information, respectively perform matching processing on the text translation result to obtain a first text translation result, where the first text translation result matches the screen display style of the current screen display content.

[0191] Through the matching process in step 1202, the text color of the first text translation result is consistent with the foreign language text in the current screen display content of the terminal and the text font size matches, and it will not affect the user's use due to the text color and text font size of the translated text. For example: the text content information (number of characters 4) is "double kill, triple kill", and after the processing in step 1202, the first text translation result (number of characters 4) is "double kill, triple kill". Therefore, the first text translation result can be set to the same text font size as the text content information.

[0192] In some embodiments, the text content information (number of characters 28) is "无双の魔呂布は言った。「これから戦場は私一人で支配します", and after the processing in step 1202, the first text translation result (number of characters 32) is "无双之魔吕布说:“从此刻开始,战场由我一人主宰!可有人敢与我一战”". Therefore, the first text translation result can be set to a font size of 5, which matches the text font size of the text content information (small size 4) to avoid ghosting due to the excessive number of characters in the first text translation result.

[0193] Step 1203: Based on the background information and foreground information corresponding to the text content information, perform rendering processing on the first text translation result to obtain the text translation result after rendering processing.

[0194] In some embodiments of the present invention, Figure 13 It is an optional flowchart of a method for translating the screen display content of a terminal provided by an embodiment of the present invention, which specifically includes the following steps:

[0195] Step 1301: Based on the background information corresponding to the text content information, configure the text region boundary detection box and the maximum search radius.

[0196] in, Figure 14 This is a diagram illustrating the display status of a virtual character in a game video. The images displayed on the mobile game screen are not static resources stored locally by the app, but rather dynamic effects generated based on user actions. As a third-party developer, it is impossible to directly obtain the original image. Furthermore, in the process of erasing the original text and restoring the original background, because there are many pixels in the text, most pixels in the text area require sufficient server participation to restore the background, and the hardware cost required for this process is too high.

[0197] Step 1302: Based on the text region boundary detection box and the maximum search radius, perform a first marking process on the pixels that exceed the text region boundary detection box to obtain a first marking result.

[0198] The maximum search radius can be adjusted according to the type of content displayed on the terminal screen or the area of ​​the display interface of the terminal device. The radius is 1 to a preset value (less than or equal to 15). When the method for translating the content displayed on the terminal screen provided in this application is triggered, the maximum search radius corresponding to the current video processing environment can be automatically selected.

[0199] Step 1303: Perform a second marking process on the pixels inside the text region boundary detection box to obtain the second marking result.

[0200] Step 1304: Traverse all the pixels corresponding to the text content information in the two-marking results and perform pixel backfilling to obtain the background processing result of the first text translation result.

[0201] The initial traversal of the bitmap aims to trade space efficiency for time efficiency by using a hash table to store the mapping between pixel coordinates and whether a pixel contains text. Pixels outside the text region boundary detection box can be directly identified as non-text regions, so they can be marked directly without color similarity comparison, and their values ​​are taken from the non-text positions.

[0202] For pixels within the detection bounding box, a basic criterion can be used to make the following judgment: the font color and the main background color have high distinguishability. Therefore, in order to mark the text position, the RGB color value of the pixel needs to be converted into an HSV color value, and then the color similarity of this value and the HSV value of the main background color is compared. If the difference exceeds a threshold, the pixel is considered to be the location of the text.

[0203] Step 1305: In response to the background processing result of the first text translation result, perform foreground information fusion processing to obtain the fused first text translation result.

[0204] Marking all pixels on the bitmap uses some memory. The purpose is to reuse the results of the local traversal during the second bitmap traversal, allowing the text pixels to be filled with a certain background color quickly enough using the marked pixels.

[0205] Specifically, such as Figure 13 As shown, during the second traversal, only all text region pixels marked in the first traversal, such as (x1, y1), are processed. Then, using this pixel (x1, y1) as the center, the surrounding pixels are traversed with a radius ranging from 1 to a preset value (where the preset value is less than or equal to 15). If a non-text pixel (x2, y2) is found, its RGB value is used to fill (x1, y1) in reverse. This process is repeated until all text pixels in all regions have been traversed. At this point, all text pixels have been filled with the closest background color in reverse.

[0206] After the background image in the video frame is fused, the foreground image of the video frame needs to be further processed to obtain the first text translation result after foreground information removal. Specifically, at this point, the foreign text on the mobile game interface on the screen has been largely removed and covered with the background color. However, there are still many noise points at the boundaries of the text content, resulting in an unattractive interface. Therefore, some morphological image processing is required on the bitmap, using erosion and dilation techniques to perform secondary processing on the color of the text area. Since the stroke thickness of the text in game videos, especially mobile game videos, varies, simple erosion and dilation processing may produce larger noise areas. Therefore, a "block-based" erosion scheme is adopted.

[0207] The image dilation calculation method refers to Formula 3, where the (x, y) surrounding region is used. Replace the value of (x, y) with the maximum value within the range.

[0208] Formula 3

[0209] The proposed technical solution does not process the color of a single pixel, but rather processes the colors of multiple pixels. In Formula 3, (x, y) represents a pixel, and src(x, y) is the pixel value. Further optimization of this solution involves taking a conceptually new pixel as a 3x3 pixel block, and normalizing the color values ​​of this block for the nine pixels it contains to obtain new pixel values. The normalization process is described in Formula 4.

[0210] Formula 4

[0211] Among them, the RGB three channels or HSV three channels of multiple pixels in the pixel block can be summed. The summation is for 9 pixels within a certain channel (such as the R channel), and the final normalization result is the average of the color values of the three channels within the pixel block as the color value of the pixel block. Through this value, the dilation operation as described above is performed.

[0212] In some embodiments of the present invention, when performing Gaussian blur processing on the image in the area where the foreground information is located, the result of the Gaussian blur processing can be converted from the RGB mode to the HSV mode to obtain the foreground information in the HSV mode; for the image in the area where the foreground information is located, morphological dilation processing and morphological erosion processing are sequentially performed to obtain the fused first text translation result. When performing morphological dilation processing, holes can be eliminated. The structural element is a rectangle with a size of 3*3. The structural element slides from left to right and from top to bottom and performs an AND operation with the image pixels within the window in turn. When at least one value of the operation result is 1, the pixel at this position is assigned 1, otherwise 0.

[0213] Finally, through the processing of steps 301-305, the text translation result matches the screen display style of the terminal screen content, and the colors and fonts are the same, which can enable the user to obtain the same usage experience as when using the terminal's current screen display content in the foreign language state.

[0214] In some embodiments of the present invention, since the image frame to be translated may be the bullet screen information in a game video, therefore, when the terminal screen display content is a game video and the text content information is a bullet screen, the content of the target bullet screen can be replaced by triggering the translation function item and using the first text translation result to form a new bullet screen. Specifically, when the original bullet screen information in the game video is "剣聖宮本武蔵は「天下無双!秘密を教えます。私は無敵です。」" in Japanese, after being processed by the translation model, the first text translation result "The sword saint Miyamoto Musashi said: "Peerless in the world! Let me tell you a secret: I am invincible!" can be presented at the display position of the original bullet screen information. Among them, the text color, font size, and display position of the presented first text translation result are the same as those of the original bullet screen information, and it will not block the playback of the game video at the same time.

[0215] In some embodiments of the present invention, when the translation results obtained from foreign-server games do not comply with relevant laws (e.g., some countries classify games based on player age) and therefore do not meet legal requirements, the game video can continue to play. However, the translation results of the video to be played and the text content currently displayed on the terminal screen need to be blurred to comply with relevant legal requirements. In some embodiments of the present invention, Gaussian blurring can be used on both the image layer and the text translation result layer to blur the video and the text translation results currently displayed on the terminal screen. The Gaussian blurring process includes: converting the available memory, decoding capability, and frame rate of the terminal's current display content of the device into four levels of information: A, B, C, and D. The Gaussian blurring degree uses the blur radius as a parameter; the higher the blur radius, the greater the performance consumption. Gaussian blurring is performed with a blur power of 25 + 5 * level to achieve different levels of blurring from 25f to 50f.

[0216] In some embodiments of the present invention, since users have different usage habits, the playback status of the video and / or the corresponding text translation results can be adjusted according to the corresponding user instructions when blurring the video and the text translation results displayed in the current screen content of the terminal.

[0217] Of course, since there are various types of terminals running video clients, and their hardware configurations are also different, different processing strategies can be implemented for devices with different configuration parameters. When the device configuration meets the configuration conditions, when translating the text content information in the image frames to be translated in the video, since the parallel processing capability of the terminal device's processor is sufficient, the method for translating the terminal screen display content provided in this application can be executed on the terminal device, without causing stuttering in video playback and / or text translation result display, allowing users to have a better viewing experience. Therefore, the terminal screen display content can continue to play, and the text translation result can be adjusted according to the corresponding background and foreground information. However, for large-scale game processes with high resolution, when the parallel processing capability of the terminal device's processor is insufficient, the method for translating the terminal screen display content provided in this application can be executed on the game server. The terminal device receives the text translation result rendered based on background and foreground information transmitted by the game server, which can also avoid stuttering in the display of text translation results. Once the game server receives the generated text translation result, it can review the result. If the review is successful, the translated text result, rendered based on background and foreground information, will be displayed on different clients (or web page playback interfaces) watching the same video. If the review fails, the translated text result, rendered based on background and foreground information, can be blurred or a prompt message can be issued to indicate that there is non-compliant information in the image frame to be translated in the content displayed on the user's terminal screen.

[0218] In some embodiments of the present invention, when the content displayed on the terminal screen is a cloud gaming video, an empirical threshold matching the image frame to be translated is determined; based on the empirical threshold matching the image frame to be translated and the boundary range of image processing matching the chroma key parameters, overflow pixels in the content displayed on the terminal screen are determined; and the grayscale values ​​of the overflow pixels in the content displayed on the terminal screen are adjusted. Specifically, by adjusting an empirical threshold P (P≥R), pixels in the original image within the closed interval [R, P] of the chroma key distance are selected, and these pixels are defined as overflow pixels. That is, some of these pixels exhibit overflow due to reflection from the screen, affecting the visual perception and requiring correction. The value of the overflow pixel can be set to a grayscale value. For example, it can be corrected according to (R+G+B) / 3. In some embodiments of the present invention, since users' perception of RGB colors is different in different usage scenarios, the weight of the overflow pixel can be flexibly adjusted according to the usage habits of different users. For example, Grey = 0.411 *R + 0.547 *G + 0.155 *B, to adapt to the usage habits of different users and make users have a more comfortable user experience.

[0219] In summary, the embodiments of the present invention have the following technical effects:

[0220] By acquiring a target video, capturing an image frame to be translated from the target video, and receiving a translation trigger command to translate the current screen display content of the terminal; triggering a translation process matching the target video, and translating the text information in the image frame to be translated through the translation process to obtain a first text translation result; responding to the translation processing command for the current screen display content of the terminal, acquiring the current screen display content of the terminal, and performing screen capture processing to obtain an image frame to be translated that includes text content information in the current screen display content of the terminal; wherein, the image frame to be translated includes text content information and text attribute information corresponding to the text content information; translating the text content information in the image frame to be translated to obtain a text translation result; acquiring the text attribute information corresponding to the text content information of the image frame to be translated, wherein, the text attribute information includes text color information and text font size information; rendering the text translation result according to the text attribute information and overlaying it onto the current screen display content page of the terminal to achieve real-time presentation of the rendered text translation result on the terminal. The text translation process involves: acquiring the text color information and text font size information corresponding to the text information; determining the background information and foreground information corresponding to the text information based on its position in the image frame to be translated; processing the text translation result according to the text color information and text font size information to obtain a second text translation result, thereby matching the second text translation result with the visual style of the target video; adjusting the second text translation result based on the background information and foreground information corresponding to the text information to integrate the second text translation result into the target video. This not only enables accurate translation of the image frame to be translated in the target video terminal display content, but also allows the text color information, text font size information, background information, and foreground information to be rendered and overlaid onto the current display content page of the terminal, matching the visual style of the terminal display content. This avoids ghosting and blurry text caused by the translation of the text content, ensuring a more comfortable user experience.

[0221] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for translating the content displayed on a terminal, characterized by, The method comprises: receiving a translation trigger instruction for translating the current screen content of the terminal; in response to the translation trigger instruction for the current screen content of the terminal, obtaining the current screen content of the terminal and performing screenshot processing to obtain an image frame to be translated including text content information in the current screen content of the terminal; wherein the image frame to be translated includes text content information and text attribute information corresponding to the text content information; performing translation processing on the text content information in the image frame to be translated to obtain a text translation result; obtaining text attribute information corresponding to the text content information of the image frame to be translated, wherein the text attribute information includes text color information and text size information; performing matching processing on the text translation result according to the text color information and the text size information in the text attribute information to obtain a first text translation result; configuring a character region boundary detection box and a maximum search radius based on background information corresponding to the text content information; performing first marking processing on pixel points outside the character region boundary detection box according to the character region boundary detection box and the maximum search radius to obtain a first marking result; performing second marking processing on pixel points inside the character region boundary detection box to obtain a second marking result; taking the pixel points corresponding to the text content information in each second marking result as the center and traversing with the maximum search radius, if the pixel points corresponding to the first marking result or the pixel points corresponding to non-text content in the second marking result are found, performing pixel reverse filling processing to obtain a background processing result of the first text translation result; performing fusion processing on the background processing result of the first text translation result and foreground information corresponding to the text content information to obtain the first text translation result after fusion.

2. The method of claim 1, wherein, The method comprises: in response to the translation trigger instruction for the current screen content of the terminal, triggering a corresponding translation model; determining at least one word-level latent variable corresponding to the text content information through the encoder of the translation model; generating a translation word corresponding to the word-level latent variable and a selected probability of the translation word according to the at least one word-level latent variable through the decoder of the translation model; selecting at least one translation word to form a text translation result corresponding to the text content information according to the selected probability of the translation result.

3. The method of claim 1, wherein, The method comprises: performing conversion processing on the format of the image frame to be translated to obtain a bitmap format image frame to be translated; performing color space conversion processing on the image frame to be translated in bitmap format, and performing identification processing on the background color in the image frame to be translated according to the result of color space conversion processing to obtain the color value of the pixel points corresponding to the background color; extracting the color value of the pixel points in the region where the text content information in the image frame to be translated is located; The color value of the pixel corresponding to the background color is compared with the color value of the pixel in the region where the text content information is located, to obtain the text color information; The text size information is obtained by segmenting the bitmap format image frame to be translated.

4. The method of claim 3, wherein, The method further comprises: segmenting the image frame to be translated according to the position of the text content information, to obtain at least two text recognition segments, counting the text size corresponding to the single character involved in the at least two text recognition segments, to obtain the text size information in each text recognition segment; comparing the text size information in different text recognition segments, and when it is determined that the horizontal starting position of different text recognition segments is less than or equal to the horizontal position threshold, and the height of the different text recognition segments is less than or equal to the height threshold, the different text recognition segments are summarized; counting the text size information of the text recognition segments obtained after the summarization, and filtering the corresponding text size information in the image frame to be translated according to the counting result.

5. The method of claim 1, wherein, The background information corresponding to the text content information and the foreground information corresponding to the text content information are determined according to the position of the text content information in the image frame to be translated; the first text translation result matches the picture style of the current terminal screen content.

6. The method of claim 5, wherein, The method further comprises: converting the image frame to be translated from a red-green-blue (RGB) mode to a hue-saturation-value (HSV) mode; cropping the text content information in the image frame to be translated according to the position of the text content information in the image frame to be translated in the HSV mode; determining the background information corresponding to the text content information according to the result of the cropping; determining the edge position of the text content information according to the result of the cropping and the position of the text content information in the image frame to be translated, and determining the foreground information according to the edge position of the text content information.

7. The method of claim 1, wherein, The background processing result of the first text translation result is fused with the foreground information corresponding to the text content information to obtain the first text translation result after fusion, which comprises: performing Gaussian blur processing on the image in the region where the foreground information is located; converting the result of the Gaussian blur processing from the RGB mode to the HSV mode to obtain the foreground information in the HSV mode; performing morphological dilation processing and morphological erosion processing on the image in the region where the foreground information is located in sequence, to obtain the text translation result after rendering.

8. The method of claim 1, wherein, The method further comprises: when the terminal screen content is a game video and the text content information is a barrage, obtaining a replacement operation for a target barrage through a barrage editing component; obtaining a second text translation result corresponding to the target barrage in response to the replacement operation; rendering the second text translation result according to the text attribute information through the barrage editing component; replacing the target barrage in the current terminal screen content page with the second text translation result after rendering.

9. The method of claim 1, wherein, The method further comprises: presenting a translation function item in the view interface, the translation function item being used to implement translation of literal information in the terminal on-screen content; in response to a triggering operation on the translation function item, obtaining and presenting the translated image frame and the corresponding background image, wherein the translated image frame includes a first text translation result.

10. The method of claim 9, wherein, The presenting a translation function item in the view interface comprises: presenting a translation function item in the form of a floating ball in the view interface, wherein the translation function item in the form of a floating ball is capable of adjusting the display position in the view interface in response to the triggering operation; or presenting a translation function item in the form of transparency in the view interface, wherein the number of pixels of the translation function item in the form of transparency is 1, and the translation function item in the form of transparency is capable of adjusting the display position in the view interface in response to the triggering operation.

11. An apparatus for translating content displayed on a terminal, characterized by The device comprises: an information transmission module configured to receive a translation triggering instruction for performing translation processing on the current on-screen content of a terminal; an information processing module configured to, in response to the translation processing instruction for the current on-screen content of the terminal, obtain the current on-screen content of the terminal and perform screenshot processing to obtain a to-be-translated image frame including text content information in the current on-screen content of the terminal; wherein the to-be-translated image frame includes text content information and text attribute information corresponding to the text content information; the information processing module is configured to perform translation processing on the text content information in the to-be-translated image frame to obtain a text translation result; the information processing module is configured to obtain text attribute information corresponding to the text content information in the to-be-translated image frame, wherein the text attribute information includes text color information and text font size information; the text translation result is matched based on the text color information and the text font size information in the text attribute information to obtain a first text translation result; a text region boundary detection box and a maximum search radius are configured based on background information corresponding to the text content information; and first marking processing is performed on pixel points that are outside the text region boundary detection box based on the text region boundary detection box and the maximum search radius to obtain a first marking result; second marking processing is performed on pixel points inside the text region boundary detection box to obtain a second marking result; and pixel reverse filling processing is performed if a pixel point corresponding to the first marking result or a pixel point corresponding to non-text content in the second marking result is found, with the pixel point corresponding to the text content information in each second marking result as the center and the maximum search radius as the radius, to obtain a background processing result of the first text translation result; the background processing result of the first text translation result is fused with foreground information corresponding to the text content information to obtain a fused first text translation result.

12. The device of claim 11, wherein The information processing module is further configured to, in response to the instruction for the translation processing of the current screen display content of the terminal, trigger a corresponding translation model; determine, by an encoder of the translation model, at least one word-level latent variable corresponding to the text content information; generate, by a decoder of the translation model, a translation word corresponding to the word-level latent variable and a selected probability of the translation word according to the at least one word-level latent variable; select at least one translation word to form a text translation result corresponding to the text content information according to the selected probability of the translation result.

13. The apparatus of claim 11, wherein The information processing module is further configured to perform a format conversion on the image frame to be translated to obtain a bitmap format image frame to be translated; perform a color space conversion on the bitmap format image frame to be translated, and identify a background color in the image frame to be translated according to a result of the color space conversion to obtain a color value of a pixel corresponding to the background color; extract a color value of a pixel in a region where the text content information is located in the image frame to be translated; compare the color value of the pixel corresponding to the background color with the color value of the pixel in the region where the text content information is located to obtain the text color information; perform a segmentation on the bitmap format image frame to be translated to obtain the text font size information.

14. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions, when executed by a processor, implement the method for the translation processing of the screen display content of the terminal according to any one of claims 1 to 10.

15. An electronic device, comprising: The electronic device comprises: a memory configured to store executable instructions; a processor configured to execute the executable instructions stored in the memory to implement the method for the translation processing of the screen display content of the terminal according to any one of claims 1 to 10.

16. A computer-readable storage medium storing executable instructions, wherein the instructions, when executed by a processor, cause the processor to perform operations comprising: The executable instructions, when executed by a processor, implement the method for the translation processing of the screen display content of the terminal according to any one of claims 1 to 10.

Images