Text translation method and apparatus, electronic device, and medium

Abstract

The present disclosure provides a text translation method and device, electronic equipment and medium, relates to the technical field of computers, in particular to the technical field of cloud mobile phones, cloud computing and cloud services. The specific implementation scheme is: rendering video stream data obtained from a cloud device to generate a current display interface; obtaining text information to be translated included in the current display interface according to a translation instruction for the current display interface, and translating the text information to be translated. The present disclosure realizes the effect of translating the rendered display interface of the cloud device, so that the user can still read smoothly when facing the multi-language content in the cloud device, improves the use experience of the cloud device, and improves the customer retention rate of the cloud device.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to the field of cloud phones, cloud computing and cloud services, and especially to a text translation method, apparatus, electronic device and medium. Background Technology

[0002] As an emerging cloud service technology, cloud devices are gaining popularity among consumers because they are not constrained by local hardware conditions.

[0003] With the development of cloud device technology, cloud devices have begun to adapt to application software in multiple languages. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and medium for translating a rendered cloud device display interface.

[0005] According to one aspect of this disclosure, a text translation method is provided, comprising:

[0006] Render the video stream data obtained from the cloud device to generate the current display interface;

[0007] According to the translation instruction for the current display interface, obtain the text information to be translated included in the current display interface, and translate the text information to be translated.

[0008] According to another aspect of this disclosure, a text translation apparatus is provided, comprising:

[0009] The display interface generation module is used to render video stream data obtained from cloud devices and generate the current display interface;

[0010] The translation module is used to obtain the text information to be translated included in the current display interface according to the translation instruction of the current display interface, and to translate the text information to be translated.

[0011] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0012] At least one processor; and

[0013] A memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in any one of the present disclosures.

[0015] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods described in any one of this disclosure.

[0016] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, represents the method described in any one of the present disclosures.

[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0018] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0019] Figure 1 This is a flowchart of some text translation methods disclosed in embodiments of this disclosure;

[0020] Figure 2 This is a flowchart of some other text translation methods disclosed according to embodiments of this disclosure;

[0021] Figure 3 These are schematic diagrams of the structures of some text translation devices disclosed in embodiments of this disclosure;

[0022] Figure 4 This is a block diagram of an electronic device used to implement the text translation method disclosed in the embodiments of this disclosure. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] During the research and development process, the applicant discovered that with the development of cloud device technology, cloud devices have begun to adapt to application software in multiple languages, such as game software, video software, and office software. Foreign game software, in particular, is very popular among young cloud device users. However, most of these games have not developed language packs adapted for the Chinese market. This leads to Chinese players experiencing difficulties due to language barriers, hindering their gaming experience and indirectly impacting the user experience of cloud devices, resulting in a decline in customer retention rates.

[0025] Figure 1 This is a flowchart of some text translation methods disclosed in embodiments of this disclosure. This embodiment can be applied to the translation of cloud device display interfaces rendered by a client. The methods in this embodiment can be executed by the text translation apparatus disclosed in embodiments of this disclosure. The apparatus can be implemented in software and / or hardware and can be integrated into any electronic device with computing capabilities.

[0026] like Figure 1 As shown, the text translation method disclosed in this embodiment may include:

[0027] S101. Render the video stream data obtained from the cloud device to generate the current display interface.

[0028] Cloud devices refer to devices that utilize cloud computing technology on the server side to provide cloud services. These devices, with their built-in systems and vendor-hosted servers, can perform numerous functions over the network. Types of cloud devices include, but are not limited to, cloud phones and cloud computers. Video stream data refers to the video frame data received by the client and generated by the cloud device based on the current interface. For example, if the current interface of the cloud device is a game interface, the video stream data consists of the video frames of each game scene; similarly, if the current interface is a movie interface, the video stream data consists of the video frames of each movie scene.

[0029] In one implementation, when a user launches any application icon on the client cloud device's display interface, such as by clicking the application icon, the client responds to the launch operation by generating a remote launch command carrying the application identifier of the target application. Here, the client refers to a smart terminal device, such as a smartphone, smartwatch, tablet, laptop, or any electronic device running a smart operating system; the application identifier, or AID (Application Identifier), is unique, meaning that each application has a unique application identifier.

[0030] The client sends a remote start command to the server. The server receives the remote start command, parses it to obtain the application identifier, and then matches the application identifier with the application identifiers of each candidate application in the cloud device. Based on the matching result, the server determines and starts the target application from the candidate applications.

[0031] After the target application starts, the server obtains the application's interface video data and sends each frame of the interface video data to the client via a video stream. After receiving the video stream data from the server, the client renders the video stream data using its graphics processor to generate the current display interface, which is then displayed on the client's display device. For example, the server might send the game interface video data of a game application, frame by frame, sequentially to the client. The client then renders each frame of the game interface video data sequentially to display the game interface in that order.

[0032] S102. According to the translation instruction for the current display interface, obtain the text information to be translated included in the current display interface, and translate the text information to be translated.

[0033] The translation command refers to the instruction to translate the text information in the currently displayed interface.

[0034] In one implementation, the user performs control operations on the client's currently displayed interface to trigger the client to generate translation instructions.

[0035] The control operations include, but are not limited to: 1. The user performs a touch operation on a preset area of ​​the current display interface, including but not limited to clicking, double-clicking, or heavy-hitting. When the client recognizes the user's touch operation, it triggers the generation of a translation command. 2. The user issues a voice control command to the client, such as "translate," "I want to see Chinese," or "translate text," etc. After receiving the voice control command, the client performs voice recognition on the voice control command and triggers the generation of a translation command based on the voice recognition result. 3. The user performs a gesture operation on the client, such as "waving," "making a fist," or "giving a thumbs up," etc. The client uses a carried image acquisition device, such as a front-facing camera, to capture the user's gesture image and matches the user's gesture image with a preset standard image. If it is determined that the user's gesture image matches the standard image, it triggers the generation of a translation command.

[0036] After receiving the translation instruction, the client generates interactive information on the current display interface, such as "Please confirm the translation area", to remind the user to confirm the translation area that needs to be translated in the current display interface.

[0037] Within a preset time threshold for generating interactive information on the current display interface, if the user selects a translation area from the current display interface based on a preset area selection operation, including but not limited to dragging a selection box to select the translation area, or manually circling the area, the client will determine the translation area from the current display interface based on the user's area selection operation. If the user does not select a translation area within the preset time threshold for generating interactive information on the current display interface, the client will default to using the entire current display interface as the translation area.

[0038] After determining the translation area, the client continues to generate interactive information on the current display interface, such as "Please select the language you want to translate into," and displays currently supported candidate languages ​​to the user. The user selects at least one candidate language as the target language, which includes, but is not limited to, Chinese, English, Korean, and Japanese. The client takes a screenshot of the translation area to generate a translation image corresponding to the translation area, and performs text recognition on the translation image based on text recognition technology, determining the text information to be translated included in the translation image based on the text recognition results. Then, the text information to be translated is translated according to the target language, including but not limited to the client uploading the text information to be translated to a translation server via network communication, translating the text information to be translated into the target language based on the computing power of the translation server, and receiving the translation result sent by the translation server; or, through a translation model pre-installed on the client, inputting the text information to be translated into a translation model, translating the text information to be translated into the target language based on the translation model, and obtaining the translation result output by the translation model.

[0039] After obtaining the translation results, the client displays the translation results on the current display interface, allowing users to read the translation results corresponding to each piece of text to be translated.

[0040] This disclosure renders video stream data obtained from a cloud device to generate a current display interface, and obtains the text information to be translated included in the current display interface according to the translation instructions for the current display interface, and translates the text information to be translated, thereby achieving the effect of translating the rendered cloud device display interface. This allows users to read smoothly when faced with multilingual content in the cloud device, improving the user experience of the cloud device and increasing the customer retention rate of the cloud device.

[0041] Figure 2 This is a flowchart of some other text translation methods disclosed in the embodiments of this disclosure, which are further optimized and extended based on the above technical solutions, and can be combined with the above optional implementation methods.

[0042] like Figure 2 As shown, the text translation method disclosed in this embodiment may include:

[0043] S201. Render the video stream data obtained from the cloud device to generate the current display interface.

[0044] S202. Based on the touch operation of the floating button in the current display interface, the translation instruction is generated.

[0045] The floating button is a pre-created function button used to trigger translation commands. Users can enable or disable the floating button function in the settings interface. When enabled, the floating button will always be displayed on the top layer of the screen, regardless of the application the user is using in the client.

[0046] In one implementation, the user pre-enables the floating button. When the user needs to translate the currently displayed interface, they perform a touch operation on the floating button, including but not limited to single-clicking, double-clicking, or long-pressing the floating button. The client then triggers the generation of translation instructions based on the user's touch operation on the floating button.

[0047] S203. Obtain a screenshot of the currently displayed interface according to the translation instruction, and perform text recognition on the screenshot to obtain the text information to be translated.

[0048] In one implementation, after the client receives the translation instruction, it generates a screenshot of the currently displayed interface based on the image data of the currently displayed interface. Then, a preset text recognition algorithm is used to perform text recognition on the screenshot to determine the text information to be translated contained within it.

[0049] Optionally, in S203, "obtaining a screenshot of the currently displayed interface according to the translation instruction" includes:

[0050] The MediaProjection interface is invoked according to the translation instructions, and a virtual screen is created through the MediaProjection interface; the image data of the currently displayed interface is obtained from the virtual screen, and the image data is rendered to obtain a screenshot of the currently displayed interface.

[0051] MediaProjection refers to an API (Application Programming Interface) used for recording or capturing screen images. Virtual display, on the other hand, is used to encode image data to generate screenshots or screen recordings.

[0052] In one implementation, the client first obtains the MediaProjectionManager object via getSystemService. Then, it calls MediaProjectionManager.createScreenCaptureIntent() to display a dialog asking the user for permission to capture the screen. If permission is granted, the MediaProjection interface is obtained.

[0053] After obtaining the MediaProjection interface, a virtual window is created using the MediaProjection interface, and parameters such as the virtual window's name, width, height, and pixel density are initialized. After initialization, the image data of the currently displayed interface is obtained through the ImageReader instruction and passed to the Surface class buffer of the virtual screen. The image data is then rendered to obtain a screenshot of the currently displayed interface.

[0054] By calling the MediaProjection interface according to the translation instructions and creating a virtual screen through the MediaProjection interface, the image data of the current display interface is obtained from the virtual screen, and the image data is rendered to obtain a screenshot of the current display interface. This achieves the effect of obtaining a screenshot of the current display interface, laying the data foundation for subsequent acquisition of the text information to be translated from the current display interface based on the screenshot.

[0055] Optionally, S203, "performing text recognition on the screenshot of the interface to obtain the text information to be translated," includes:

[0056] The screenshot of the interface is converted to a bitmap format, and the converted screenshot is input into a text recognition model. Based on the output of the text recognition model, the text information to be translated in the screenshot and the location information associated with the text information to be translated are obtained.

[0057] Bitmap format is the format used in this context.

[0058] In one implementation, the screenshot is converted from its original image format to a bitmap format using pre-set image processing software on the client side. This image processing software includes, but is not limited to, Photoshop, Painter, and Adobe Illustrator. The converted screenshot is then input into a pre-built text recognition model on the client side. This text recognition model is an OCR (Optical Character Recognition) model, which includes a convolutional neural network model. The screenshot is then subjected to OCR by the text recognition model. Based on the output of the text recognition model, the text information to be translated and its corresponding location information are obtained. The location information refers to the position of the text information to be translated within the image area of ​​the screenshot.

[0059] By converting the screenshot to bitmap format and inputting the converted screenshot into a text recognition model, the model obtains the text information to be translated from the screenshot, as well as the location information associated with the text information, based on the output of the text recognition model. This achieves the effect of recognizing the text information to be translated from the screenshot based on a standard format. Furthermore, since the text recognition model is built locally on the client, it does not need to rely on a server over the network for recognition, avoiding the problem of slow or unrecognizable recognition due to large network latency, thus improving the reliability and timeliness of text recognition.

[0060] S204. Input the text information to be translated into the translation model, and obtain the standard text information corresponding to the text information to be translated based on the output of the translation model.

[0061] Here, "translation model" refers to a machine translation model. In this embodiment, the translation model includes, but is not limited to, a convolutional neural network translation model, and optionally a seq2seq model.

[0062] In one implementation, the acquired text information to be translated is input into a pre-established translation model on the client side, and then the translation model is controlled to translate the text information to be translated according to the target language. Based on the output of the translation model, the standard text information in the target language form corresponding to the text information to be translated is obtained.

[0063] S205. Generate a standard text image corresponding to the standard text information, and overlay the standard text image onto the interface screenshot according to the location information associated with the text information to be translated.

[0064] In one implementation, the standard text information is drawn into a standard text image in image format by using a preset drawing technique. For example, the standard text information is drawn into a standard text image through the canvas technique, and according to the determined position information associated with the text information to be translated in the interface screenshot, the standard text image is overlaid on the position corresponding to the position information in the interface screenshot. For example, assume that the text information to be translated is "apple", its associated position information in the interface screenshot is (X, Y), and the standard text information corresponding to "apple" is "苹果". Then the standard text image corresponding to "苹果" is overlaid on the (X, Y) position in the interface screenshot.

[0065] The present disclosure triggers the generation of a translation instruction according to the touch operation on the floating button in the current display interface, realizes the quick generation of the translation instruction, and improves the translation efficiency; by obtaining the interface screenshot of the current display interface according to the translation instruction and performing text recognition on the interface screenshot to obtain the text information to be translated, it realizes the effect of determining the text information to be translated by text recognition of the interface screenshot, laying a data foundation for subsequent translation; by inputting the text information to be translated into the translation model and obtaining the standard text information corresponding to the text information to be translated according to the output result of the translation model, it realizes the effect of determining the standard text information through the translation model. Since the translation model is established locally on the client, there is no need to rely on the server through the network for text translation, avoiding the problem of slow text translation or inability to perform text translation due to large network latency, and improving the reliability and timeliness of text translation; by generating the standard text image corresponding to the standard text information and overlaying the standard text image on the interface screenshot according to the position information associated with the text information to be translated, the user can more intuitively obtain the standard text information. Since the standard text image is in the same position as the text information to be translated, it does not affect the normal viewing of the interface screenshot by the user.

[0066] Figure 3 It is a schematic structural diagram of some text translation devices disclosed according to an embodiment of the present disclosure, which is applicable to the situation of translating the cloud device display interface rendered by the client. The device in this embodiment can be implemented by software and / or hardware and can be integrated on any electronic device with computing capabilities.

[0067] As Figure 3 shown, the text translation device 30 disclosed in this embodiment may include a display interface generation module 31 and a translation module 32, where:

[0068] The display interface generation module 31 is configured to render the video stream data obtained from the cloud device to generate the current display interface;

[0069] The translation module 32 is used to obtain the text information to be translated included in the current display interface according to the translation instruction of the current display interface, and to translate the text information to be translated.

[0070] Optionally, the device further includes a translation instruction generation module, specifically used for:

[0071] The translation command is generated based on the touch operation of the floating button in the current display interface.

[0072] Optionally, the translation module 32 is specifically used for:

[0073] According to the translation instruction, obtain a screenshot of the currently displayed interface, and perform text recognition on the screenshot to obtain the text information to be translated.

[0074] Optionally, the translation module 32 is further used for:

[0075] The MediaProjection interface is invoked according to the translation instructions, and a virtual screen is created through the MediaProjection interface;

[0076] The image data of the current display interface is obtained from the virtual screen, and the image data is rendered to obtain a screenshot of the current display interface.

[0077] Optionally, the translation module 32 is further used for:

[0078] The screenshot of the interface is converted to a bitmap format, and the converted screenshot is then input into the text recognition model.

[0079] Based on the output of the text recognition model, obtain the text information to be translated in the screenshot of the interface, as well as the location information associated with the text information to be translated.

[0080] Optionally, the translation module 32 is further used for:

[0081] The text information to be translated is input into the translation model, and the standard text information corresponding to the text information to be translated is obtained based on the output of the translation model.

[0082] Optionally, the device further includes an image overlay module, specifically used for:

[0083] Generate a standard text image corresponding to the standard text information, and overlay the standard text image onto the interface screenshot according to the location information associated with the text information to be translated.

[0084] The text translation apparatus 30 disclosed in this embodiment can execute the text translation method disclosed in this embodiment, and has the corresponding functional modules and beneficial effects for executing the method. Content not described in detail in this embodiment can be referred to the description in the method embodiments of this disclosure.

[0085] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0086] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0087] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0088] like Figure 4 As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0089] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0090] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as text translation methods. For example, in some embodiments, the text translation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the text translation method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform text translation methods by any other suitable means (e.g., by means of firmware).

[0091] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0092] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0093] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0094] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0095] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0096] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0097] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0098] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A text translation method, comprising: rendering video stream data obtained from a cloud device to generate a current display interface; capturing a gesture image of a user and matching the gesture image with a standard image, and if it is determined that the gesture image matches the standard image, triggering a translation instruction for the current display interface; obtaining text information to be translated included in the current display interface according to the translation instruction for the current display interface, and translating the text information to be translated; wherein the obtaining of the text information to be translated included in the current display interface according to the translation instruction for the current display interface comprises: generating interactive information in the current display interface according to the translation instruction to remind the user to determine a translation region in the current display interface that needs to be translated; if it is determined that the user selects a translation region from the current display interface based on a preset region selection operation within a preset time threshold of the generation of the interactive information in the current display interface, determining the translation region from the current display interface according to the region selection operation of the user; if it is determined that the user does not select a translation region within the preset time threshold of the generation of the interactive information in the current display interface, taking the entire current display interface as the translation region; capturing a screenshot of the translation region to generate a translation image corresponding to the translation region, and performing text recognition on the translation image based on a text recognition technology, and determining the text information to be translated included in the translation image according to a text recognition result; wherein the translating of the text information to be translated comprises: uploading the text information to be translated to a translation server through network communication, translating the text information to be translated into a target language based on the computing power of the translation server, and receiving a translation result issued by the translation server; wherein the capturing of the screenshot of the translation region to generate the translation image corresponding to the translation region comprises: obtaining a MediaProjectionManager through getSystemService to obtain a MediaProjectionManager object; calling MediaProjectionManager.createScreenCaptureIntent() to pop up a dialog to ask the user whether to authorize the application to capture the screen, and if the authorization is successful, obtaining a MediaProjection interface; creating a virtual window through the MediaProjection interface, and initializing name, width, height and pixel density parameters of the virtual window; after initialization, obtaining image data of the translation region through an ImageReader instruction, and inputting the image data into a Surface class buffer of the virtual screen, and then rendering the image data to obtain a translation image of the translation region.

2. The method of claim 1, before the obtaining of the text information to be translated included in the current display interface according to the translation instruction for the current display interface, further comprising: The translation command is generated based on the touch operation of the floating button in the current display interface.

3. The method of claim 1, wherein, The step of obtaining the text information to be translated included in the current display interface according to the translation instruction of the current display interface includes: According to the translation instruction, obtain a screenshot of the currently displayed interface, perform text recognition on the screenshot, and obtain the text information to be translated.

4. The method of claim 3, wherein, Perform text recognition on the screenshot to obtain the text information to be translated, including: The screenshot of the interface is converted to a bitmap format, and the converted screenshot is then input into the text recognition model. Based on the output of the text recognition model, the text information to be translated in the screenshot of the interface, as well as the location information associated with the text information to be translated, are obtained.

5. The method of claim 4, wherein, The translation of the text information to be translated includes: The text information to be translated is input into the translation model, and the standard text information corresponding to the text information to be translated is obtained based on the output of the translation model.

6. The method according to claim 5, after obtaining the standard text information corresponding to the text information to be translated, further includes: Generate a standard text image corresponding to the standard text information, and overlay the standard text image onto the interface screenshot according to the location information associated with the text information to be translated.

7. A text translation device, comprising: The display interface generation module is used to render video stream data obtained from cloud devices and generate the current display interface; The translation module is used to collect the user's gesture image and match the gesture image with a standard image. If it is determined that the gesture image matches the standard image, a translation command for the currently displayed interface is generated. According to the translation instructions for the current display interface, obtain the text information to be translated included in the current display interface, and translate the text information to be translated; Specifically, the translation module is used for: Based on the translation instructions, interactive information is generated on the current display interface to remind the user to identify the translation area that needs to be translated on the current display interface; If, within a preset time threshold for generating interactive information on the current display interface, it is determined that the user selects a translation region from the current display interface based on a preset region selection operation, then the translation region is determined from the current display interface according to the user's region selection operation. If it is determined that the user has not selected a translation area within the preset time threshold for generating interactive information on the current display interface, then the entire current display interface will be used as the translation area. A screenshot of the translation region is taken to generate a translation image corresponding to the translation region. Text recognition technology is used to perform text recognition on the translation image, and the text information to be translated included in the translation image is determined based on the text recognition results. Specifically, the translation module is further used for: The text information to be translated is uploaded to the translation server via network communication. Based on the computing power of the translation server, the text information to be translated is translated into the target language, and the translation result sent by the translation server is received. The translation module is specifically further configured to: Obtain a MediaProjectionManager, and obtain a MediaProjectionManager object through getSystemService; Call MediaProjectionManager.createScreenCaptureIntent() to pop up a dialog to ask the user whether to authorize the application to capture the screen, and if the authorization is successful, obtain a MediaProjection interface; Create a virtual window through the MediaProjection interface, and initialize the name, width, height, and pixel density parameters of the virtual window; After the initialization, obtain image data of the translation area through an ImageReader instruction, and input the image data into a Surface class buffer of a virtual screen, so as to render the image data and obtain a translation image of the translation area.

8. The apparatus of claim 7, further comprising a translation instruction generation module, specifically configured to: Trigger generation of the translation instruction according to a touch operation on a floating button in the current display interface.

9. The apparatus of claim 7, wherein, The translation module is specifically configured to: Obtain an interface screenshot of the current display interface according to the translation instruction, and perform text recognition on the interface screenshot to obtain the text information to be translated.

10. The apparatus of claim 9, wherein, The translation module is specifically further configured to: Convert the interface screenshot into a bitmap format, and input the interface screenshot in the bitmap format into a text recognition model; According to an output result of the text recognition model, obtain the text information to be translated in the interface screenshot and position information associated with the text information to be translated.

11. The apparatus of claim 10, wherein, The translation module is specifically further configured to: Input the text information to be translated into a translation model, and according to an output result of the translation model, obtain standard text information corresponding to the text information to be translated.

12. The apparatus of claim 11, further comprising an image covering module, specifically configured to: Generate a standard text image corresponding to the standard text information, and cover the standard text image to the interface screenshot according to the position information associated with the text information to be translated.

13. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-6.

14. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to execute the method of any one of claims 1-6.

15. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-6.

Images